Slow parity check-Turn off Docker, speeds up

[poopsie]

Just as the title says, my parity check is really slow if I have dockers runnings.  I have an R710, Dual Xeon x5570 with 5 drives (4 x 4TB, 1 x 2TB in Dual parity Config).  This is my first parity check using the server, and the numbers were slow - Around 30MBs.  I turned off all the dockers, and the numbers on each drive went up to 140MBs.  The average CPU is 1%-3%, so that is not the issue.  My transfer speeds are 115 MBs, so I don't think there is a bottleneck with the H200 card.  My dockers that I am running are

 

Crashplan

Resilio

Plex

Krusader

 

No VM's on, 16GB of ECC RAM.  Does anyone have any insight as to why this would be?  Is there a script to turn off all dockers before starting and restarting them after parity check is done?

 

Thanks!!

Edited May 2, 2018 by poopsie

[John_M]

Presumably your dockers are accessing the array while the parity check is underway, but that's just a guess as you've provided no information to work with.

[poopsie]

4 hours ago, John_M said:

Presumably your dockers are accessing the array while the parity check is underway, but that's just a guess as you've provided no information to work with.

 

Whatever I need to provide to help I will.  What exactly are you looking for?

 

 

[John_M]

Tools -> Diagnostics. Post the zip file. Make sure you haven't rebooted since the problem showed up because rebooting discards the logs.

 

If it turns out that your dockers are accessing the array during the parity check maybe you ought to reschedule the latter. Normally people want to optimise their server with the services it provides in mind, rather than switching off services in order to speed up the parity check. If I want to watch a Plex movie during a parity check then it's much preferable for the parity check to be slowed down than for me to have to read a book instead due to Plex being unavailable!

 

[poopsie]

2 hours ago, John_M said:

Tools -> Diagnostics. Post the zip file. Make sure you haven't rebooted since the problem showed up because rebooting discards the logs.

 

If it turns out that your dockers are accessing the array during the parity check maybe you ought to reschedule the latter. Normally people want to optimise their server with the services it provides in mind, rather than switching off services in order to speed up the parity check. If I want to watch a Plex movie during a parity check then it's much preferable for the parity check to be slowed down than for me to have to read a book instead due to Plex being unavailable!

 

 

No reboot since then-I appreciate you taking a look!!

unraid-diagnostics-20180502-1445.zip

[John_M]

Your parity check starts at midnight and at 02:00 Community Apps Backup and Restore stops your docker containers (cadvisor, CrashPlanPRO, Krusader, plex, resilio-sync), backs up their appdata and verifies the backup, then restarts the containers at 02:19. So you might want to schedule the backups to start earlier and the parity check to start later to avoid this clash.

 

The syslog is quiet for the rest of the morning. At 12:23 there's a burst of network activity from a docker (I'm not sure whether it's plex or resilio-sync). At 18:00 something creates a new user share called Downloads and there's more docker activity. At 22:19 there's a lot of docker activity for 10 minutes. At 22:40 disk3 spins down (it's your smallest so it's no longer needed by the parity check). The dockers are quiet now and your cache is rested, woken and rested again. Midnight passes and at 01:15 the parity check is complete. unRAID tries to connect to gmail to send you a confirmation email and fails. The remaining disks spin down at 01:30 and the system rests until being woken up at 02:00 again by Community Apps Backup and Restore.

 

It isn't possible to see what dockers are doing - reads/writes aren't logged - but the system seems to be working without any obvious errors. Of your dockers I'm familiar only with Krusader and plex. Krusader is a graphical file manager and basically does nothing unless you interact with it. Plex, on the other hand, runs all sorts of background processes under the guise of "the butler", scanning your media collection and downloading metadata, indexing and cross-referencing, in addition to it's obvious job of serving up media. I understand cadvisor monitors and gathers metric on other containers, while CrashPlan is a cloud backup/sync utility and resilio sync is a pee-to-peer sync tool, so I suppose these latter two are likely to be involved in accessing your array.

 

I hope this gives you a few clues.

[kizer]

I have my CA backup set for the 2nd of every month myself just to insure this doesn't happen. 

[poopsie]

15 hours ago, John_M said:

Your parity check starts at midnight and at 02:00 Community Apps Backup and Restore stops your docker containers (cadvisor, CrashPlanPRO, Krusader, plex, resilio-sync), backs up their appdata and verifies the backup, then restarts the containers at 02:19. So you might want to schedule the backups to start earlier and the parity check to start later to avoid this clash.

 

The syslog is quiet for the rest of the morning. At 12:23 there's a burst of network activity from a docker (I'm not sure whether it's plex or resilio-sync). At 18:00 something creates a new user share called Downloads and there's more docker activity. At 22:19 there's a lot of docker activity for 10 minutes. At 22:40 disk3 spins down (it's your smallest so it's no longer needed by the parity check). The dockers are quiet now and your cache is rested, woken and rested again. Midnight passes and at 01:15 the parity check is complete. unRAID tries to connect to gmail to send you a confirmation email and fails. The remaining disks spin down at 01:30 and the system rests until being woken up at 02:00 again by Community Apps Backup and Restore.

 

It isn't possible to see what dockers are doing - reads/writes aren't logged - but the system seems to be working without any obvious errors. Of your dockers I'm familiar only with Krusader and plex. Krusader is a graphical file manager and basically does nothing unless you interact with it. Plex, on the other hand, runs all sorts of background processes under the guise of "the butler", scanning your media collection and downloading metadata, indexing and cross-referencing, in addition to it's obvious job of serving up media. I understand cadvisor monitors and gathers metric on other containers, while CrashPlan is a cloud backup/sync utility and resilio sync is a pee-to-peer sync tool, so I suppose these latter two are likely to be involved in accessing your array.

 

I hope this gives you a few clues.

 

Wow, that is an amazing report!  Thanks!

 

13 hours ago, kizer said:

I have my CA backup set for the 2nd of every month myself just to insure this doesn't happen. 

 

I think between the two of you, I have decided to loosen up a few things on my dockers (Plex being one, not as much updating) and keeping everything running with a slower parity.  John nailed it on the head, I would rather have a slower parity, but still have access to everything. 

 

Thanks guys!!

[kizer]

I have my machine to maximize parity checks meaning I have things scheduled on the Second and if I have anything scheduled and running with the User.Scripts Plugin I have them set to not run during a parity check. However I can't stop the family from watching videos via Plex when they want to. 

 

Sure Fast Parity Checks are kinda cool, but lets face it if we stress out about our servers constantly then they are no longer fun. 

[poopsie]

1 minute ago, kizer said:

I have my machine to maximize parity checks meaning I have things scheduled on the Second and if I have anything scheduled and running with the User.Scripts Plugin I have them set to not run during a parity check. However I can't stop the family from watching videos via Plex when they want to. 

 

Sure Fast Parity Checks are kinda cool, but lets face it if we stress out about our servers constantly then they are no longer fun. 

 

You and John totally changed my mindset on that.  Unraid will do what is has to do, just slower and still provide everything I need.  Thanks!!

[John_M]

4 hours ago, kizer said:

Sure Fast Parity Checks are kinda cool, but lets face it if we stress out about our servers constantly then they are no longer fun. 

 

That sums it up really well, kizer!