Mover / Parity Check very slow

[Recoil]

Hi folks, I tried to move data from cache to the array with mover and it was very slow, I check the forums and the advise was to do a "mover stop" and use "Dynamix File Manager" I did but still the write speeds where crawling. 

I ended up moving the important files and deleting the other files since it was taking too long, changed the settings to not use the cache since is so slow that it is unusable.

While I was working on this I had a power failure and I had a dirty restart, Parity Check kicked in but it did say that it will take something like 20 days or so to complete.

That was last night, I checked this morning finish moving/deleting the files from the cache. The parity check still was showing days to complete  however, I checked again once I was done deleting/moving files and now it says 9 hours that is more about what I usually get.

I am unsure what is going on, I am attaching my "diagnostics" could you folks please advise? Thank you!

ark-diagnostics-20230721-0728.zip

[itimpi]

If you are moving files and running a parity check at the same time they will both be significantly slowed down as you end up getting disk contention.   It is far more efficient to run these operations separately.

[Recoil]

I stopped mover last night no longer it was operating, only the parity check and it was still showing days to complete the parity check.

I am unsure what is going on.

[JorgeB]

There's something writing to disk1.

[Recoil]

Thank you for the replies guys, please correct me if I am wrong, if something is writing to one disk it will slow down parity check that much? 

I am guessing when it stopped writing to disk1 the parity check went back to normal this morning? I mean it went from something like 20 days to 16 hours now.

Also please let me ask you about cache, "mover" was moving large files to the array but it was very slow I left it moving overnight and it seems like no progress was done.

Is that normal? I was thinking since cache is an NVMe SSD files could be written faster there and later on transfer to the array at least at the same speed as if I was writing directly to the array however, it seems that it is extremely slow to move files from the cache.

I am wondering if the purpose of cache is to be used with VM's and containers not to cache files before sending to the array? Thank you once more for the advise.

[JorgeB]

8 minutes ago, Recoil said:

Thank you for the replies guys, please correct me if I am wrong, if something is writing to one disk it will slow down parity check that much? 

Yes, because it's also writing to parity.

[JorgeB]

9 minutes ago, Recoil said:

Also please let me ask you about cache, "mover" was moving large files to the array but it was very slow I left it moving overnight and it seems like no progress was done.

Is that normal? I was thinking since cache is an NVMe SSD files could be written faster there and later on transfer to the array at least at the same speed as if I was writing directly to the array however, it seems that it is extremely slow to move files from the cache.

It should be the same, nothing else writing to the array at that time?

[Recoil]

Nothing else was writing other than the mover at the time I was testing it.

[itimpi]

The problem is that any writes to the array also require writes to the corresponding sector on the parity drive.   You are thus getting constant head movement back and forth on the parity drive between the sector(s) involved in the write and the sectors being accessed by the current position of the parity check process.  Since head movements are comparitively slow this leads to very significant slow downs of both processes.

[Recoil]

I see, so here is the million dollar question, should cache be enabled? 

It seems it is going to be extremely slow if there is other activity on the array would there be any pros to have it enabled?

[itimpi]

Just now, Recoil said:

I see, so here is the million dollar question, should cache be enabled?

The whole idea is that writes to the cache are independent of the main array and so run at the full speed the cache can support.   The (much slower) transfer of files from cache to array by mover are normally scheduled to run in the middle of the night when the server is typically idle.

[Recoil]

Yes that makes sense 100% however, please correct me if I am wrong since I think I was trying to do too much in one go I did not know the limitations of the array.

I tried to copy to the array with cache enabled about 2000 files that totaled about 800GB, if memory serves me well they copy in a few hours to cache. I expected to take at least twice as much to copy to the array with mover so I did let it run overnight and I think it did probably copy under 10% to the array.

If I am following what you say, when copying large amounts of data to the array I should do it without cache? 

On the other hand for everyday use I should leave cache enabled? Once more thank you for all the advise.

 

[itimpi]

1 minute ago, Recoil said:

If I am following what you say, when copying large amounts of data to the array I should do it without cache? 

On the other hand for everyday use I should leave cache enabled? Once more thank you for all the advise.

Yes.

[Recoil]

Thank you!

[Recoil]

Hi guys hope you can help I am a bit stomped I thought this issue was squashed by not using the cache.

I added the Turbo write plugin and I tried to transfer the data directly to the array (no cache) and it did work fine I was seeing about 90mb/s on speed.

Today I tried to send another batch started fine but eventually it slow down to kilobits as far as I can tell nothing else is writing to disk I do not have VMs or containers so I do not know what could be slowing down my transfers.

I cancel the transfer and did give it a try just now it did the same started up to over 100mb/s but eventually it went down to kilobits. 

I can see in the dashboard how 2 - 3 cores at a time in the CPU max out to 100% here and there stay there for a bit and come down, it keeps cycling different cores. Usually I do not see that I do not know if that is related.

I am attaching diagnostics from this morning, hope you guys can point me in the right direction.

 

ark-diagnostics-20230724-1140.zip

[JorgeB]

Disks 1 and 2 are SMR, disk1 model especially have been known to be extra slow sometimes, if you limit writes to disk3 is it normal?

[Recoil]

Hi Jorge, you be correct. Disk 3 works fine all the time, it does not slow down over time. Can I ask you how do you know what disks are SMR, thank you!

[JorgeB]

23 minutes ago, Recoil said:

Can I ask you how do you know what disks are SMR

I know most models, but some have that info in the SMART report:

Model Family:     Seagate BarraCuda 3.5 (SMR)

 

 

 

[Recoil]

Good to know, thank you!