Large copy between shares never finishes

[SciKo]

I've ran into something I quite don't understand and was hoping maybe someone here sits on some knowledge regarding the matter.

I've been trying to complete my server migration by copying large chunks of backed up data from an isolated share (it's disk will be made into a parity disk when done). This has worked great seeing the last job copy ~3.5TB. This last job however I chose to copy the remaining ~7.15TB in one go and it has shown some strange behaviour which I can't make out if it's purely UI or something else going on.

Specifically after quite some time in - the progress and transfer speed does not update in the webGUI even though the animated circle icon keeps spinning suggesting the UI hasn't frozen completely.


Multiple times I haven't waited this stall out and eventually, hours later it starts updating again but last time I noticed it regressed from 100% to 92% (or likely much less but I noticed it at 92%). During these stalls I have noticed the disks are still reading and writing as shown in the disk array view so I've assumed it's just a disconnect between the UI and the actual copy job. Now I'm not so sure anymore and am starting to worry if I'm stuck in a loop and need to cancel/redo?

I'm using the built in copy feature for clarity.

 

Does anyone have an idea what this phenomenon is?

 

 

Diagnostics from today during the screenshotted stall at 100% provided.
 

skynet-diagnostics-20230727-1534.zip

[dlandon]

What share are you copying to?  I only see the isos share set up for a cache.

 

Set up all your shares with the cache.

 

Also, turn off mover logging.  It can flood the syslog making it hard to read.

[trurl]

21 minutes ago, dlandon said:

Set up all your shares with the cache.

 

1 hour ago, SciKo said:

copy the remaining ~7.15TB in one go

I doubt their cache is that large. I always say if you need to transfer more than cache will hold, don't cache. And mover only makes things worse, it is intended for idle time.

[SciKo]

Yeah I temporarily turned off the cache for the two shares in question (shares: server-migration -> media) since my cache is only 1TB. My intention is then to have a cache -> array setup for the media share.

 

As of just now it looks like the copy transaction is finally finished although it still confuses me greatly what caused it to prolong for roughly 36 hours.
I will take your advice and turn off mover logging. As for this type of stalling I'm wondering if there can be some hardware level caching going on in the mechanical HDDs that could stall once the supposed cache is full?

It will probably be quite a while until I make another copy transaction this large but I'd nevertheless like to know what went wrong or was overlooked this time.

[dlandon]

2 hours ago, SciKo said:

Yeah I temporarily turned off the cache for the two shares in question (shares: server-migration -> media) since my cache is only 1TB. My intention is then to have a cache -> array setup for the media share.

 

As of just now it looks like the copy transaction is finally finished although it still confuses me greatly what caused it to prolong for roughly 36 hours.
I will take your advice and turn off mover logging. As for this type of stalling I'm wondering if there can be some hardware level caching going on in the mechanical HDDs that could stall once the supposed cache is full?

It will probably be quite a while until I make another copy transaction this large but I'd nevertheless like to know what went wrong or was overlooked this time.

There is ram disk caching that can collect a lot of information and then writes to the disk which can look like it holds back the transfer.  That cache can be by default very large.  You have a lot of ram.  Install the Tips and Tweaks plugin and adjust it to see if it can help. I don't see this situation causing a 36 hour slow down though.

 

You might want to look at some other tweaks in the Tips and Tweaks plugin like on your NICs.

[SciKo]

50 minutes ago, dlandon said:

You might want to look at some other tweaks in the Tips and Tweaks plugin like on your NICs.

Thank you I'll take a look at that!
In this particular instance (copy between shares via SATA) the NIC shouldn't matter though right?

[dlandon]

1 hour ago, SciKo said:

Thank you I'll take a look at that!
In this particular instance (copy between shares via SATA) the NIC shouldn't matter though right?

No.  I thought you were doing a network copy.  Disk to disk should be affected by NIC speed.