Mover/array speed suddenly seems diminished

[Puregreen59]

so im very new to unraid/linux

 

and ive been having having fun setting everything up, i recently added a NVME drive to help with Sabnzbd and overall system response, for the most part its been good, but today i've had some strange behavour where mover seems to either get stuck or Moves VERY slowly from the cache drive to the array, while its moving from the cache drive to the array it seems to Write at lets say 5mb to drive 4, but it also reads from drive 4 at 5mb and read and writes the same to parity, from my experience so far with unraid, that seams strange/odd, ive had to restart (uncleanly ill add) twice now one of witch took ssh'ing to force a reboot as it seemed stuck

 

could any one shine some light on what could be happening?

jarvis-diagnostics-20210808-0148.zip

[trurl]

Why is your system share cache-yes? It should be cache-prefer or only so dockers/VM stay on cache for performance and don't keep array disks spunup.

 

Your appdata share is cache-only, but it has some files on array. Mover ignores cache-only shares, you have to set that to cache-prefer to get mover to move it to cache where it belongs.

 

domains is often kept on cache also and yours is cache-prefer so that is OK.

 

You have a few other shares besides those that are set to cache-prefer. Why do you want those to stay on cache?

 

It looks like you may have accidentally created some folders at the top level of pool or array disks. Any folder at the top level of pools or array is automatically a User Share.

 

Do you still seem to have problems if you remove Mover Tuning plugin?

[Puregreen59]

OK so i seem to have found Turbo write on the forum.. 😅 and doing some reading that would point to why its reading and writing like it is..? im currently letting it do its long overdue parity cheak to make sure everything is OK after its mishaps.

 

i had noticed evan in parity cheak it was going SLOWW, appon enabling turbo write, it went up to 40mb's and slowly up to 160 at top end, and bounces around betweenn 40 - 140mb's

 

can someone just comfirm everything looks normal there end?

 

i guess i never saw this problem till now because i've only just got everything back online, as in bringing up sabnzbd sonnar, radarr and lidarr. im assuming the cure to this would be a bigger cache drive so then I wouldn't be forced to empty out my 500gb (375gb after cache only's) like 8 times in one day 

 

Thank you for pointing that out Trurl

 

righhttt! Cache  ONLY is what i want for system your very right! 

 

i've got a few others on cache just to speed up acsess over network while i was filling them up to be honest, i do need to go over them with a fine toothpick really.

[trurl]

Turbo write will have no effect on parity check speed since all disks are already being read simultaneously during parity check.

 

Parity check speed should settle down after a short time, and be somewhat faster in the beginning than at the end due to faster outer cylinders at the beginning and slower inner cylinders at the end. This will also be affected by the specific disks still involved in parity check. In addition to different RPM for different disk models, smaller disks will be finished first and not be used while larger disks continue the check.

[Puregreen59]

its done it again, i thought i had solved it, so its been running its parity cheak, woke up to find it doing a 318kb parity cheak it was doing about 140mb before i went to sleep..but yeah.. with 281 days Remaining 😂

 

[Puregreen59]

right so im a little comfused, im assuming it has somthing to do with one of my dockers..?? i disabled docker to do some testing, Started a extended SMART test on every drive, and it was still going super slow, after about 2 or 3 min's it sped back up 

[ChatNoir]

52 minutes ago, Puregreen59 said:

its done it again, i thought i had solved it, so its been running its parity cheak, woke up to find it doing a 318kb parity cheak it was doing about 140mb before i went to sleep..but yeah.. with 281 days Remaining 😂

I suppose at least one drive or controller is not behaving correctly.

 

Your diagnostics during the check might show something.

[Puregreen59]

6 minutes ago, ChatNoir said:

I suppose at least one drive or controller is not behaving correctly.

 

Your diagnostics during the check might show something.

well i hope you know more than the uninitiated that i am  Haha! i paused the parity check to try and get to the bottom of it, but on the assumption that the log has info from this morning when i work up maybe there's something interesting in this one.

jarvis-diagnostics-20210808-1033.zip

[Puregreen59]

so just a update, i noticed that stopping all my dockers didn't do it today, then i remembered that i have a VM ill just add that System, iso's and domain are all cache ONLY so they shouldn't have been writing to the array and have confirmed they is none of them shares on the Disk1, Dist2 ect ect.

 

its all on cache where i expect it to be, any ideas on this weird behaviour? 

 

could it be the cache drive?.. i don't know its really strange.. i mean its a old nvme ssd but it should still be WAY good enough to at the very least be able to do more than 5mb's while its running a VM that's idle? i feel like its not that, i feel like something strange, either something that's normal with unraid i don't know about, or something bugging out.

 

some background specs, 

asus tuff x570 plus

ryzen 3900X

32gb ram

IBM ServeRAID M5110 flashed in IT mode (LSI SAS2308)

the Cache drive is a Samsung 950 PRO 512gb (BTFS)

5 x 8tb Seagate baracuda SMR (5400 i think? all shucked, one of these is my parity drive)

1 x 6tb Seagate baracuda SMR (5400 i think? also shucked)

Nvidia 1080 

[Puregreen59]

wait hold on hold on, 

 

i found a script on the forum for auto mover at a user defined Percentage, is that perhaps messing things up.

 

because now I think about it, i think its every time ITS ran to empty out the cache that this happens.

 

Also I noticed that no where in the script mentions turbo write, will it just use the default turbo write that is turned on in disk settings, or do I need to define it in this script, and if so how would i add it?

 

 

 

#!/usr/bin/php
<?PHP

$moveAt = 68;    # Adjust this value to suit.

$diskTotal = disk_total_space("/mnt/cache");
$diskFree = disk_free_space("/mnt/cache");
$percent = ($diskTotal - $diskFree) / $diskTotal * 100;

if ( $percent > $moveAt ) {
  exec("/usr/local/sbin/mover");
}
?>

 

 

Thanks for your help guys, i know it must be frustrating to help along us noobs 

[Puregreen59]

I can confirm that I have all my dockers and VM currently running, and downloading on sabnzbd, mover isn't running and nothing appears to be accessing the drives to write, as i would expect.

 

and i have a parity check running again with Expected speeds. on the next cache fill up ill invoke mover Manually and see if i get the same behaviour

[trurl]

According to that last diagnostics you posted, appdata still has files on disk1. Mover can't move open files so you will have to go to Settings and disable Docker before those can be moved to cache. Mover also won't move duplicates, so if you have the same files on cache and disk1 for appdata then you will have to sort that out manually.

 

As for Mover Tuning plugin, I don't use it. Maybe you should reconsider how you are using cache.

 

It is impossible to move from fast cache to slower array as fast as you can write to cache. And if you are trying to move while also writing then these are just going to compete for the same resources. Mover is intended for idle time. If you need to write more at one time than cache can hold, don't cache.

 

Also, you don't want to ever let cache fill up or you will likely corrupt it.

 

Each pool (cache) has a Minimum Free setting. You should set that to larger than the largest file you expect to write.

 

Unraid has no way to know how large a file will become when it chooses a disk for it. If cache has less than Minimum, Unraid will choose an array disk instead of cache for cache-prefer or cache-yes shares (overflow). Cache-only shares it can only choose cache even if it is full.

 

Each User Share also has a Minimum Free setting that works in a similar manner. If a disk has less than Minimum, Unraid will choose a different disk when it begins writing the file.

 

Note that once a disk is chosen for a file, if it runs out of space you will get an error.

[trurl]

And just to give some idea of what is typical for parity check, I usually estimate 2-3 hours per TB of parity. So for you, 16-24 hours for 8TB parity.

 

Also, nothing has changed about this problem I mentioned earlier:

16 hours ago, trurl said:

It looks like you may have accidentally created some folders at the top level of pool or array disks. Any folder at the top level of pools or array is automatically a User Share.

 

Go to Shares - User Shares, click Compute All, wait for the result, then post a screenshot.

 

[Puregreen59]

 

 

 

 

5 hours ago, trurl said:

 

Go to Shares - User Shares, click Compute All, wait for the result, then post a screenshot.

 

 

I've crossed one out, don't ask questions  and  I wont lie too you 🤣  

[Puregreen59]

okay so im still having trouble with with mover. i've currently almost finished migrating to unraid, and ive set up my lists on radarr and theres quite a back log off movies, a good tb or 2, that need to be downloaded un-rared, recoded to h265 via Tdarr while on the cache drive then move to the array, but what's holding me up now is this mover problem.

 

if anyone can shed some light on more commands or something im missing please enlighten me. im lost and a bit frustrated because i know what happens next,

 

from here im gunna get bored of waiting for seemingly nothing, im going to try and reboot unraid, that's gunna take forever, possibly fail, but if it doesn't fail its going to reboot and tell me it detected a unclean shutdown. then i've got a 50/50% chance its gunna work perfectly fine and move everything like nothing was wrong. i really need some in-sight to whats going on here.

 

hopefully this helps show what i see my side as well as logs.. 

 

jarvis-diagnostics-20210904-2120.zip

[Puregreen59]

thought id do a video of the reboot procress, forgot to hit record a little earlier, i disabled docker and vm manager to help aid a faster reboot.

 

 

so after rebooting its perfectly fine as i thought it would be! 🤣😢