Buffering issues while running parity check


Recommended Posts

I actually experienced something that I have not seen on previous RCs/betas...

 

Running a parity check caused the blu-ray rip I was watching to buffer pretty consistently (gbit connection).  I'll do some more testing to see if I can reproduce and nail down the exact circumstances.  The easiest slution for me at the time was to cancel the parity check to keep the kiddos happy watching their movie.

 

In the meantime others may want to test.

 

John

Link to comment

Over lunch yesterday, using plex web to watch a small tv show, it was stuttering badly, never seen this before, had no time to see what was going on. Will re-test later. Via a mac mini running plex client had no issue watching the same shows later that evening. I never had issues with the Plex web (via browser) so will try again over lunch today. No parity check was running. Disk tunables adjusted.

 

Tom is already aware, that during a parity check with the latest RCs (will retest for RC16c but we both know there will be no difference) you can barely make connects to unRAID via AFP clients. Stated this would be worked on post 5.0 unfortunately.

Link to comment

I've actually seen buffering while streaming HD content a few times recently during parity checks, and I've never seen this before...

 

Note: Gigabit wired connections and I have custom set tunables.

 

Edit: Streaming to an XBMC box using SMB

Link to comment

I confess I'm a little surprised by this thread.  A parity check/build is one of the most I/O taxing jobs a computer can possibly do, essentially maximizing I/O bandwidth usage.  It stuffs the I/O pipes as hard as it can.  You would normally never want to run one while using your server, for any uploads or downloads but especially not while streaming.  It's a very demanding maintenance task, that you would normally only run in off hours when there is no demand for the server, and then only perform it as infrequently as you feel is safe.

 

In fact, I find it rather remarkable that anyone has ever streamed successfully during a parity check.  I suppose it is possible, if they are well buffered at the playback machine.

 

We've recently (hopefully) seen the last of the uproar over parity check speeds, with many users wanting the very fastest possible speeds for their parity checks.  It's been hard for me not to think the uproar somewhat overblown, since a parity check is just an off-hour maintenance task that is normally done only when little else would be running.  (I do realize some of the emphasis on the parity check was more likely using it as an overall performance metric.)  But we have seen parity check speeds now reach near maximums, and it sounds like users are now starting to complain about the opposite, that it is going so fast it is slowing down streaming.  I suspect that any users who thought they had glitch-free streaming before, perhaps had factors that were slowing down their parity check speeds such that the interruptions (re-buffering and head movement) caused by streaming I/O were not noticeable.

 

What you may be asking Tom for here, is to provide a way to throttle parity checks back, give streaming processes a higher priority, essentially asking Tom to slow the checks down!  I suspect he is not going to be thrilled, after spending all this time to eliminate the various factors limiting its speed.  I suppose it's normal human nature, "it's too fast", it's too slow", "why can't we run everything at the same time?", etc.

Link to comment

I confess I'm a little surprised by this thread.  A parity check/build is one of the most I/O taxing jobs a computer can possibly do, essentially maximizing I/O bandwidth usage.  It stuffs the I/O pipes as hard as it can.  You would normally never want to run one while using your server, for any uploads or downloads but especially not while streaming.  It's a very demanding maintenance task, that you would normally only run in off hours when there is no demand for the server, and then only perform it as infrequently as you feel is safe.

 

In fact, I find it rather remarkable that anyone has ever streamed successfully during a parity check.  I suppose it is possible, if they are well buffered at the playback machine.

 

We've recently (hopefully) seen the last of the uproar over parity check speeds, with many users wanting the very fastest possible speeds for their parity checks.  It's been hard for me not to think the uproar somewhat overblown, since a parity check is just an off-hour maintenance task that is normally done only when little else would be running.  (I do realize some of the emphasis on the parity check was more likely using it as an overall performance metric.)  But we have seen parity check speeds now reach near maximums, and it sounds like users are now starting to complain about the opposite, that it is going so fast it is slowing down streaming.  I suspect that any users who thought they had glitch-free streaming before, perhaps had factors that were slowing down their parity check speeds such that the interruptions (re-buffering and head movement) caused by streaming I/O were not noticeable.

 

What you may be asking Tom for here, is to provide a way to throttle parity checks back, give streaming processes a higher priority, essentially asking Tom to slow the checks down!  I suspect he is not going to be thrilled, after spending all this time to eliminate the various factors limiting its speed.  I suppose it's normal human nature, "it's too fast", it's too slow", "why can't we run everything at the same time?", etc.

 

Amen.  I only wish that I had been able to express myself so well regarding this 'issue'! 

Link to comment

I suspect he is not going to be thrilled, after spending all this time to eliminate the various factors limiting its speed.  I suppose it's normal human nature, "it's too fast", it's too slow", "why can't we run everything at the same time?", etc.

 

Quite the contrary

 

I think what’s  happening is that parity-check is dominating the i/o subsystem to the extent that netatalk operations slow to a crawl.  The linux md-driver has a concept of “speed limit” that does a couple of things.  First you are able to define min and max parity check/sync rates in terms of MB/sec.  If the rate starts going above the max then parity sync/check is delayed.  I don’t think the community is going to like that one very much.  The other thing it does is implement a notion of “business” in the i/o subsystem.  As requests start coming in to the md-driver any outstanding parity sync/check is slowed down to let the normal I/O finish faster.  I ripped all this out years ago but I think it should go back in.

 

I personally believe with the amount of disks we can use, no matter how fast the parity checks can be it still takes X amount of time to complete and there will be time one needs access. AFP for one is getting crushed. So I rather have the parity check slow down IF one needs access, as stated in the quote above.

 

So +1 for Tom to get this code back in, now that it is believed we got the parity check process to its full potential. Again parity will run full speed if nothing needs access, but will throttle down so we can when we need access, knowing it will take longer while we are accessing it, versus sorry to bad i am running a parity check "under construction come back later".

 

 

 

Link to comment

I confess I'm a little surprised by this thread.  A parity check/build is one of the most I/O taxing jobs a computer can possibly do, essentially maximizing I/O bandwidth usage.  It stuffs the I/O pipes as hard as it can.  You would normally never want to run one while using your server, for any uploads or downloads but especially not while streaming.  It's a very demanding maintenance task, that you would normally only run in off hours when there is no demand for the server, and then only perform it as infrequently as you feel is safe.

 

In fact, I find it rather remarkable that anyone has ever streamed successfully during a parity check.  I suppose it is possible, if they are well buffered at the playback machine.

 

We've recently (hopefully) seen the last of the uproar over parity check speeds, with many users wanting the very fastest possible speeds for their parity checks.  It's been hard for me not to think the uproar somewhat overblown, since a parity check is just an off-hour maintenance task that is normally done only when little else would be running.  (I do realize some of the emphasis on the parity check was more likely using it as an overall performance metric.)  But we have seen parity check speeds now reach near maximums, and it sounds like users are now starting to complain about the opposite, that it is going so fast it is slowing down streaming.  I suspect that any users who thought they had glitch-free streaming before, perhaps had factors that were slowing down their parity check speeds such that the interruptions (re-buffering and head movement) caused by streaming I/O were not noticeable.

 

What you may be asking Tom for here, is to provide a way to throttle parity checks back, give streaming processes a higher priority, essentially asking Tom to slow the checks down!  I suspect he is not going to be thrilled, after spending all this time to eliminate the various factors limiting its speed.  I suppose it's normal human nature, "it's too fast", it's too slow", "why can't we run everything at the same time?", etc.

 

Very well said.  While it's possible to do other things during a parity check, if you think about the head movement that causes, the thrashing of the disks is clearly going to slow things down.  If the player app has good buffering, it should be able to play a stream or two without issue;  but it makes far more sense to simply do parity checks as overnight tasks when the server is otherwise idle.    I definitely don't think Tom needs to spend time "tweaking" this aspect of the system ... he's already provided the key "tunables" that let users adjust the resources dedicated to disk buffers.  But the best way to "tune" your parity check is to simply let it run undisturbed.    If you start a parity check and later decide you need to stream a couple movies, the best option is to simply abort the check and run it again later  :)

Link to comment

If you start a parity check and later decide you need to stream a couple movies, the best option is to simply abort the check and run it again later  :)

Yes that is the current workaround, though I've been able to stream to an xbmc client just fine during all three cases: parity check, parity sync, data rebuild.  The linux 'md' driver implements a scheme where they monitor non-parity-operation disk I/O, and if they see any, they introduce a 500ms delay in the loop generating the parity stripe operations.  Not sure if this is effective, seems like it should be and not that hard for me to put in, but I'm not touching the driver until there's a 5.0 stable release out.

Link to comment

I don't think it's about head movement and thrashing. two different reads from two different processes is not thrashing. It's about balance with read tasks and buffering. The drives are designed to be used, not babied. People generally state that a parity check is the most taxing task. It's not. it's a gentle sequential read from every drive.

 

The issue I've seen with the latest releases is that when a parity check is running, everything else becomes slow. I believe it has something to do with the amount of i/o scheduling and buffering. Perhaps there are tunables that can be adjusted in the md driver that allow the read tasks more room to read. Or lower the md-sync window. I don't think you can have it all though.

 

I've used my unraid server while parity checks and generates were running. it was slower, but not unusable.

With the latest release, it's 'almost' unusuable.

 

Drives are now faster, there are more of them and we demand more out of the software using them.

 

I would suggest possibly copying the large blue ray file to the cache drive and doing a parity check to see if it's interference at the controller level or with the parity check it self.

 

I would suggest this also.  play the blue ray file.

Then

 

copy another large blueray file on the same drive to /dev/null in the background 

 

i.e.

cp file /dev/null &

 

perhaps throw another in the background until you start to see skipping.

Link to comment

If you start a parity check and later decide you need to stream a couple movies, the best option is to simply abort the check and run it again later  :)

Yes that is the current workaround, though I've been able to stream to an xbmc client just fine during all three cases: parity check, parity sync, data rebuild.  The linux 'md' driver implements a scheme where they monitor non-parity-operation disk I/O, and if they see any, they introduce a 500ms delay in the loop generating the parity stripe operations.  Not sure if this is effective, seems like it should be and not that hard for me to put in, but I'm not touching the driver until there's a 5.0 stable release out.

 

Now that sounds like a nice tuntable.

XBMC has always worked fine for me too.

Link to comment

The linux 'md' driver implements a scheme where they monitor non-parity-operation disk I/O, and if they see any, they introduce a 500ms delay in the loop generating the parity stripe operations.  Not sure if this is effective, seems like it should be and not that hard for me to put in

500ms is huge.  It would be nice if you make it adjustable through sysctl.

 

 

I know when I used to use linux software raid1 it was a big help.

The raid1 sync would slow down as I used the system.

 

I also remember there was a minimum and maximum throughput parameter in /proc.

 

It was cool as it would let me tune according to the machine's capability and my usage pattern.

 

If i was going to use the machine allot I would set the maximum MB/s to a certain value the machine would tolerate.

If I was going to bed I would bump it up.

 

The auto tune/delay was neat, but I found myself retuning the minimum and maximum myself.

 

Link to comment

If you start a parity check and later decide you need to stream a couple movies, the best option is to simply abort the check and run it again later  :)

Yes that is the current workaround,...

Come on really? I have a truck that does 0-60 in 6 seconds. But when I add 4 passengers and cargo it can't do those numbers, I understand that when I loaded up the truck. If it wouldn't drive at all though, I would be like wtf right? Or tell everyone get out and toss the cargo so I can start... workaround.

 

... though I've been able to stream to an xbmc client just fine during all three cases: parity check, parity sync, data rebuild.

Now u sound like the old man poster here, if it works for me its good for all.. and I know you know thats not the case.

 

  The linux 'md' driver implements a scheme where they monitor non-parity-operation disk I/O, and if they see any, they introduce a 500ms delay in the loop generating the parity stripe operations.  Not sure if this is effective, seems like it should be and not that hard for me to put in, but I'm not touching the driver until there's a 5.0 stable release out.

This should be added to the next RC, but if Final is hampering something then update hdparm/smartmontools and release this RC16d as final already. With a 5.1 Beta1 at the same time with various other stuff. What you find from 5.1 betas you could selectively issue patches for 5.0

As i don't see a plugin manager, afp, and other various fixes coming to 5.0 at this point. If I am wrong, sorry ahead of time.

 

Link to comment

Come on really? I have a truck that does 0-60 in 6 seconds. But when I add 4 passengers and cargo it can't do those numbers, I understand that when I loaded up the truck. If it wouldn't drive at all though, I would be like wtf right? Or tell everyone get out and toss the cargo so I can start... workaround.

4 passengers and cargo does not a load make.  Load that truck bed full of sand or attach a 25 foot trailer to it and then show me it doing 0-60 in 6 seconds.

 

 

A parity check might be "easy" on the disks but it is by no means easy on the I/O system and the disk buffers.  With a constant inflow of reads they have to to be flushed and filled.  Managing to curb the constant quick inflow of a parity check while a movie might also be playing may not be an easy task.

Link to comment

Come on really? I have a truck that does 0-60 in 6 seconds. But when I add 4 passengers and cargo it can't do those numbers, I understand that when I loaded up the truck. If it wouldn't drive at all though, I would be like wtf right? Or tell everyone get out and toss the cargo so I can start... workaround.

4 passengers and cargo does not a load make.  Load that truck bed full of sand or attach a 25 foot trailer to it and then show me it doing 0-60 in 6 seconds.

Clearly you miss read my comment.

 

But what you're saying then is all this time work has been put in to give party checks maximum priority and allow it to saturate the entire system thereby nothing else would work. Tom was loading my bed full of sand or attaching a 25 foot trailer so that if I need to play media, do critical backups, have mover work, I should cancel the parity check.

 

I didn't +1 for that, as a matter of fact I never complained about parity checks as I felt it takes as long as it needs to, it doesn't run every night and I was able to work off the server while it was in progress.

 

You guys have him spooked to change things of value so take RC16c and final it and get this over with.

Link to comment

But what you're saying then is all this time work has been put in to give party checks maximum priority and allow it to saturate the entire system thereby nothing else would work. Tom was loading my bed full of sand or attaching a 25 foot trailer so that if I need to play media, do critical backups, have mover work, I should cancel the parity check.

 

I didn't +1 for that, as a matter of fact I never complained about parity checks as I felt it takes as long as it needs to, it doesn't run every night and I was able to work off the server while it was in progress.

Actually I haven't changed anything in the parity sync/check code for a long time.  All I did was document the use of the 'md_xxx' tuning parameters.  Just as you can increase queuing to increase parity sync/check performance, you can also go the other way and scale it back so it has less impact on current I/O.

 

You guys have him spooked to change things of value so take RC16c and final it and get this over with.

Peace man, I'm not easily spooked.  There are other reasons why I have to get to a 5.0 stable release.  I'm well aware of some problems with AFP (as are other NAS vendors btw), but I've made the decision to defer this relatively major upgrade until after 5.0 stable is released.

Link to comment

But what you're saying then is all this time work has been put in to give party checks maximum priority and allow it to saturate the entire system thereby nothing else would work. Tom was loading my bed full of sand or attaching a 25 foot trailer so that if I need to play media, do critical backups, have mover work, I should cancel the parity check.

 

I didn't +1 for that, as a matter of fact I never complained about parity checks as I felt it takes as long as it needs to, it doesn't run every night and I was able to work off the server while it was in progress.

Actually I haven't changed anything in the parity sync/check code for a long time.  All I did was document the use of the 'md_xxx' tuning parameters.  Just as you can increase queuing to increase parity sync/check performance, you can also go the other way and scale it back so it has less impact on current I/O.

I haven't change anything either :) but a lot has change since I started with 5.0 Beta's (this is not an argument) I don't know what the 'md_xxx' tuning parameters are, I can ask offline, no worries.

 

You guys have him spooked to change things of value so take RC16c and final it and get this over with.

Peace man, I'm not easily spooked.  There are other reasons why I have to get to a 5.0 stable release.  I'm well aware of some problems with AFP (as are other NAS vendors btw), but I've made the decision to defer this relatively major upgrade until after 5.0 stable is released.

I know you have your reasons why and definitely not questioning them, all i am saying is if bringing this to finally as it is now, then just do it.

6 disk low power SMB users will be happy (sarcasm)

Link to comment

A parity check might be "easy" on the disks but it is by no means easy on the I/O system and the disk buffers.  With a constant inflow of reads they have to to be flushed and filled.  Managing to curb the constant quick inflow of a parity check while a movie might also be playing may not be an easy task.

 

i couldn't agree more.  Yet I do remember linux raid1 working really well with the automated background tuning.

I'm sure it can be done, although I would agree with Tom about not touching the md driver at this point in time.

Once 5.0 is out the door... well it's stretch those fingers, brighten the monitors and fully caffeinate !!! There's work to be done!! 

Link to comment

Pardon me for butting in here, but let me propose a different way of handling this problem.

 

What parity check needs is suspend/resume capability.  Once a parity check has been running for a long time, the last thing one wants to do is stop it and only to have to start over again at the beginning. 

 

Once parity check has suspend/resume capability, controlling it with a scheduler would be the next logical step.  Being able to schedule parity checks to run only during time periods which would not impact daily/normal unRAID activity would be a big plus.

 

But please, let's get 5.0 final out the door first.  This can wait. 8)

Link to comment

I can live with the option of running parity checks during off hours.  However, my original point simply was that I never experienced this behavior with previous betas/RCs.  It could have just been the fact that the movie I was streaming was on a "suspect" disk or something else...I have yet to verify.  Again...not a big deal for me but it may be for a handful of users out there (namely if they have jobs other than movie streaming running in parallel with a parity check).

 

My vote:  move forward with the other (more prevalent) outstanding issues to get v5 out the door.

 

John

 

Link to comment

I can live with the option of running parity checks during off hours.  However, my original point simply was that I never experienced this behavior with previous betas/RCs.  It could have just been the fact that the movie I was streaming was on a "suspect" disk or something else...I have yet to verify.  Again...not a big deal for me but it may be for a handful of users out there (namely if they have jobs other than movie streaming running in parallel with a parity check).

 

My vote:  move forward with the other (more prevalent) outstanding issues to get v5 out the door.

 

John

 

+1! This is why I posted my observation as well. I certainly don't want to slow down the release with more tweaks. Something has changed though and it may be worth looking into for 5.1 etc...

 

Also, some great ideas in this thread! Suspend/resume and/or throttling when other tasks are running are both very interesting...

Link to comment

What you may be asking Tom for here, is to provide a way to throttle parity checks back, give streaming processes a higher priority, essentially asking Tom to slow the checks down!  I suspect he is not going to be thrilled, after spending all this time to eliminate the various factors limiting its speed.  I suppose it's normal human nature, "it's too fast", it's too slow", "why can't we run everything at the same time?", etc.

 

In theory this sounds like a great idea.  But then when we take into the consideration the pool of testers that we have available.....  If we can't get people to run without plugins before posting problems then how are we going to get them to stop accessing their server before they start whinging that parity checks are slow again.....

Link to comment
  • 3 weeks later...
  • 2 weeks later...

Today I was doing a rebuild and streamed 1080p over gig-e smb without a single issue.

 

Not all 1080p is the same.  I've had this issue myself on some files and not others.  You need to take a look at the combined bitrate of the audio and video, higher rate files will have the problem and others will not.

Link to comment