[SOLVED] Very slow mover and netlink errors

[JonathanM]

7 minutes ago, mbc0 said:

if you can think of anything else I can provide to diagnose this please please let me know!

 

10 minutes ago, jonathanm said:

Can you access the Unraid GUI from another device temporarily and disconnect the network from your main PC?

 

[JorgeB]

5 hours ago, mbc0 said:

This is with the router switched off for 1 minute, nearly 80mb/s !

Note that the total read speed is this high because you have turbo write and it reads all disks when there are array writes, you just need to find who/what is doing that, what @jonathanmsuggested is the first thing to try, you can also boot with GUI mode to still have the GUI after disconnecting the server form the LAN.

[mbc0]

1 hour ago, johnnie.black said:

Note that the total read speed is this high because you have turbo write and it reads all disks when there are array writes, you just need to find who/what is doing that, what @jonathanmsuggested is the first thing to try, you can also boot with GUI mode to still have the GUI after disconnecting the server form the LAN.

Ok, that makes sense, this morning just checking now I can see there are 9 disks reading/writing at different speeds and 11 not being written to or read from.  Am I right in that Turbo write is not active at this moment and they are 9 seperate writes?

 

To make sure I am on the right path,

 

when the VM & Docker services are off, there is still a lot of activity, the only way to stop that activity is to boot into safe mode without the plugins running. My Desktop PC is the only other machine on the lan that can access the shares and I am certainly not doing that.  If you look at the second screen shot below, those are the open files I believe would be the cause of the activity shown in the 1st screen shot and this is what is confusing me

 

/mnt/user/Downloads is an old share that has not been used for a week or so, after following your advice Downloads are now going to an unassigned device. 

 

/mnt/user/DataToSort

/mnt/user/Games

/mnt/user/processingfolder 

These shares are all storage folders, I have not accessed them for weeks, nothing needs to access them, only me when I want something from them

 

Is there any other way I can find out what is calling on these shares when I know it is not me, there is nothing else on this network (no laptops, Pc's just phones, tv's satellite receivers, google homes etc)

 

Also  I see there are 2 process id's accessing the same shares? does this mean that there are 2 instances accessing shares?

 

[mbc0]

@johnnie.black

 

Unfortunately the slow speeds have returned, I am downloading from the internet faster than my cache can write back to the array and the mover is never finishing, the cache is just filling up, I can only now presume I have a serious hardware/config issue as my 10 year old microserver which is now running with usb drives (due to backplane failure) and a standard ssd drive cache is 50 times faster than this.  Before I (somehow) start a complete rebuild from scratch is there anything else you can think of that I can do?  I should be able to move from an NVME cache to SAS controlled SAS drives faster than 1mb/s.

 

Thank you

[JorgeB]

Did you find out and stopped what's doing the multiple writes to the array? Until you fix that there's not much point in looking at other things.

[mbc0]

Just now, johnnie.black said:

Did you find out and stopped what's doing the multiple writes to the array? Until you fix that there's not much point in looking at other things.

I have given you all the information you have asked for and I can gather from the forums, I just do not know what else to do other than start from scratch.  I cannot find what is causing these writes only that it is not anything on my network, docker or VM's the only way I can stop the writes is to boot into safe mode.

[JorgeB]

If it doesn't do it in safe mode remove all plugins, check it still doesn't do it then start adding them one by one.

[mbc0]

ok, 

 

I guess there isn't a way to just stop the plugin service so if I remove all the plugins will I lose all my plugin configs/settings or are they stored elsewhere?

[JorgeB]

You can just rename the current plugins so they don't install on boot, you can then rename them back, they are on flash/config/plugins, change the extension from .plg to anything else.

[mbc0]

11 minutes ago, johnnie.black said:

You can just rename the current plugins so they don't install on boot, you can then rename them back, they are on flash/config/plugins, change the extension from .plg to anything else.

thank you

 

[mbc0]

Hi @johnnie.black

 

I have tried with all my plugins deleted, docker disabled & vm manager disabled but there is still write activity on lots of disks even with all shares set to use cache.  What I have noticed is that on start of array with everything enabled there is no array activity at all, just cache. it is about 5 minutes before the activity starts so i have attached 3 diags.  1, immediately after array start, 2, about 1 minute after everything started, 3, about 6 minutes after array start when the activity started again.  Can you please take a look to see if you can see a difference (if it works like that) or try to see if you can see anything that is causing this behaviour making my server almost unusable 😞  

 

Thank you!

unraidserver-diagnostics-20200426-0223.zip unraidserver-diagnostics-20200426-0221.zip unraidserver-diagnostics-20200426-0227.zip

Edited April 26, 2020 by mbc0

[JorgeB]

What if you disconnect the LAN cable?

[mbc0]

18 minutes ago, johnnie.black said:

What if you disconnect the LAN cable?

That is how I know it is not my network, I disconnected the lan and checked activity over a new unused connection / subnet

[JorgeB]

This is a weird one.

 

Lets first confirm if it's related to something installed/configured on the flash drive, backup current flash drive, recreate it using the USB tool, then restore only super.dat and the key (both on the config folder), boot Unraid, disable docker and VM services, start array, check for disk activity.

[mbc0]

21 hours ago, johnnie.black said:

This is a weird one.

 

Lets first confirm if it's related to something installed/configured on the flash drive, backup current flash drive, recreate it using the USB tool, then restore only super.dat and the key (both on the config folder), boot Unraid, disable docker and VM services, start array, check for disk activity.

I have a positive update! I have half fixed it!

 

After some detective work and ALOT of moving data around I found that the one thing in common with all the disks being written to was my music share.  One by one I moved disk to disk (30+ hours) to one directory after setting the share to use disk 18.  The end result is that I now have performance back! but )there is a but) disk 18 is constantly being written to despite the share (as all shares) are set to use cache, so "half fixed"  It's great that my server is now useable but what on earth is writing to my music share with all dockers, network, vm's disabled?

 

 

 

 

 

[mbc0]

Hi @johnnie.black

 

Just an update and to thank you for all your time!

 

The reason so many disks were being written to was because my music folders were scattered across them.

 

The reason they were constantly being written to was because of a user script I wrote years ago (and forgot about) that changed the permissions every 5 minutes.

 

Again, massive thanks!