Recoil Posted July 21, 2023 Share Posted July 21, 2023 Hi folks, I tried to move data from cache to the array with mover and it was very slow, I check the forums and the advise was to do a "mover stop" and use "Dynamix File Manager" I did but still the write speeds where crawling. I ended up moving the important files and deleting the other files since it was taking too long, changed the settings to not use the cache since is so slow that it is unusable. While I was working on this I had a power failure and I had a dirty restart, Parity Check kicked in but it did say that it will take something like 20 days or so to complete. That was last night, I checked this morning finish moving/deleting the files from the cache. The parity check still was showing days to complete however, I checked again once I was done deleting/moving files and now it says 9 hours that is more about what I usually get. I am unsure what is going on, I am attaching my "diagnostics" could you folks please advise? Thank you! ark-diagnostics-20230721-0728.zip Quote Link to comment
itimpi Posted July 21, 2023 Share Posted July 21, 2023 If you are moving files and running a parity check at the same time they will both be significantly slowed down as you end up getting disk contention. It is far more efficient to run these operations separately. Quote Link to comment
Recoil Posted July 21, 2023 Author Share Posted July 21, 2023 I stopped mover last night no longer it was operating, only the parity check and it was still showing days to complete the parity check. I am unsure what is going on. Quote Link to comment
JorgeB Posted July 21, 2023 Share Posted July 21, 2023 There's something writing to disk1. Quote Link to comment
Recoil Posted July 21, 2023 Author Share Posted July 21, 2023 Thank you for the replies guys, please correct me if I am wrong, if something is writing to one disk it will slow down parity check that much? I am guessing when it stopped writing to disk1 the parity check went back to normal this morning? I mean it went from something like 20 days to 16 hours now. Also please let me ask you about cache, "mover" was moving large files to the array but it was very slow I left it moving overnight and it seems like no progress was done. Is that normal? I was thinking since cache is an NVMe SSD files could be written faster there and later on transfer to the array at least at the same speed as if I was writing directly to the array however, it seems that it is extremely slow to move files from the cache. I am wondering if the purpose of cache is to be used with VM's and containers not to cache files before sending to the array? Thank you once more for the advise. Quote Link to comment
JorgeB Posted July 21, 2023 Share Posted July 21, 2023 8 minutes ago, Recoil said: Thank you for the replies guys, please correct me if I am wrong, if something is writing to one disk it will slow down parity check that much? Yes, because it's also writing to parity. Quote Link to comment
JorgeB Posted July 21, 2023 Share Posted July 21, 2023 9 minutes ago, Recoil said: Also please let me ask you about cache, "mover" was moving large files to the array but it was very slow I left it moving overnight and it seems like no progress was done. Is that normal? I was thinking since cache is an NVMe SSD files could be written faster there and later on transfer to the array at least at the same speed as if I was writing directly to the array however, it seems that it is extremely slow to move files from the cache. It should be the same, nothing else writing to the array at that time? Quote Link to comment
Recoil Posted July 21, 2023 Author Share Posted July 21, 2023 Nothing else was writing other than the mover at the time I was testing it. Quote Link to comment
itimpi Posted July 21, 2023 Share Posted July 21, 2023 The problem is that any writes to the array also require writes to the corresponding sector on the parity drive. You are thus getting constant head movement back and forth on the parity drive between the sector(s) involved in the write and the sectors being accessed by the current position of the parity check process. Since head movements are comparitively slow this leads to very significant slow downs of both processes. Quote Link to comment
Recoil Posted July 21, 2023 Author Share Posted July 21, 2023 I see, so here is the million dollar question, should cache be enabled? It seems it is going to be extremely slow if there is other activity on the array would there be any pros to have it enabled? Quote Link to comment
itimpi Posted July 21, 2023 Share Posted July 21, 2023 Just now, Recoil said: I see, so here is the million dollar question, should cache be enabled? The whole idea is that writes to the cache are independent of the main array and so run at the full speed the cache can support. The (much slower) transfer of files from cache to array by mover are normally scheduled to run in the middle of the night when the server is typically idle. Quote Link to comment
Recoil Posted July 21, 2023 Author Share Posted July 21, 2023 Yes that makes sense 100% however, please correct me if I am wrong since I think I was trying to do too much in one go I did not know the limitations of the array. I tried to copy to the array with cache enabled about 2000 files that totaled about 800GB, if memory serves me well they copy in a few hours to cache. I expected to take at least twice as much to copy to the array with mover so I did let it run overnight and I think it did probably copy under 10% to the array. If I am following what you say, when copying large amounts of data to the array I should do it without cache? On the other hand for everyday use I should leave cache enabled? Once more thank you for all the advise. Quote Link to comment
itimpi Posted July 21, 2023 Share Posted July 21, 2023 1 minute ago, Recoil said: If I am following what you say, when copying large amounts of data to the array I should do it without cache? On the other hand for everyday use I should leave cache enabled? Once more thank you for all the advise. Yes. Quote Link to comment
Recoil Posted July 24, 2023 Author Share Posted July 24, 2023 Hi guys hope you can help I am a bit stomped I thought this issue was squashed by not using the cache. I added the Turbo write plugin and I tried to transfer the data directly to the array (no cache) and it did work fine I was seeing about 90mb/s on speed. Today I tried to send another batch started fine but eventually it slow down to kilobits as far as I can tell nothing else is writing to disk I do not have VMs or containers so I do not know what could be slowing down my transfers. I cancel the transfer and did give it a try just now it did the same started up to over 100mb/s but eventually it went down to kilobits. I can see in the dashboard how 2 - 3 cores at a time in the CPU max out to 100% here and there stay there for a bit and come down, it keeps cycling different cores. Usually I do not see that I do not know if that is related. I am attaching diagnostics from this morning, hope you guys can point me in the right direction. ark-diagnostics-20230724-1140.zip Quote Link to comment
Solution JorgeB Posted July 31, 2023 Solution Share Posted July 31, 2023 Disks 1 and 2 are SMR, disk1 model especially have been known to be extra slow sometimes, if you limit writes to disk3 is it normal? Quote Link to comment
Recoil Posted July 31, 2023 Author Share Posted July 31, 2023 Hi Jorge, you be correct. Disk 3 works fine all the time, it does not slow down over time. Can I ask you how do you know what disks are SMR, thank you! Quote Link to comment
JorgeB Posted July 31, 2023 Share Posted July 31, 2023 23 minutes ago, Recoil said: Can I ask you how do you know what disks are SMR I know most models, but some have that info in the SMART report: Model Family: Seagate BarraCuda 3.5 (SMR) Quote Link to comment
Recoil Posted July 31, 2023 Author Share Posted July 31, 2023 Good to know, thank you! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.