Server I/O Problems since 6.7.0

[Maticks]

Hi All,

 

I know some changes took place in the Kernel during the 6.7.0 Update in regards to I/O.

I run Plex on Ethernet from my Unraid Server to my TV and also Direct Play so no Transcoding is taking place.

If Mover is running the Video Pauses every few seconds till it stops.

I have setup Mover Tuner to try and lower the Priority of Mover.

I thought it did work but it turns out it hasn't.

I tried using reconstruction write instead of auto to see if turbo would fix it, but nope it doesn't seem to be a write speed issue.

Plex is also running out of ram for /tmp.

my SSD is an NVME Drive so its not a speed issue on the SSD either.

My settings are attached. So i decided to try rolling back to before 6.7.0 and after a decent amount of testing everything works flawlessly.

 

Is there anyway to run 6.7.2 with whatever changed before 6.7.0 that broke I/O when mover is running ?

 

Seems to be a common recently topic we are all having the same issue, so something has changed in the last release update.

 

 

 

Edited August 1, 2019 by Maticks

[Froberg]

I have noticed the Plex issues, too, actually. Thought it might be a connectivity issue, rather than a server issue. Just hadn't linked them to the mover process until now.

 

Edit; I've managed to confirm the issue with Plex failing to start videos is also mover-related.

Mover process appears to have slowed down, too. I tried setting it to normal priority in the tuner utility to do my test..

Edited August 3, 2019 by Froberg

[Froberg]

I think the issue extends further. I'm doing a rebuild now, due to having to replace a failing drive, something I have done with UnRaid before. So far it's run for 24 hours and I haven't seen it get higher than 50MB/s. Usually it's in the 20's.

That's significantly slower than I've seen before in this setup.

Might be related.

 

Here's how it's looking 23 hours in:

 

And here's the parity check history..

 

 

Fairly consistent, IMO.

Edited August 4, 2019 by Froberg
added pictures for clarification

[Marshalleq]

23 hours ago, Froberg said:

I think the issue extends further. I'm doing a rebuild now, due to having to replace a failing drive, something I have done with UnRaid before. So far it's run for 24 hours and I haven't seen it get higher than 50MB/s. Usually it's in the 20's.

That's significantly slower than I've seen before in this setup.

Might be related.

 

Here's how it's looking 23 hours in:

 

And here's the parity check history..

 

 

Fairly consistent, IMO.

I think you mean in the high 120's?  Yes I've noticed too.  I went down the array tuning path, but really that's not the issue.  Linux is supposed to be more performant than other OS's when it comes to file transferring, but lately it's less.  And that damn Plex issue - I've put the mover on minimum everything, but we shouldn't have to - there's clearly some kernel I/O scheduling issue.

[Froberg]

4 hours ago, Marshalleq said:

I think you mean in the high 120's?  Yes I've noticed too.  I went down the array tuning path, but really that's not the issue.  Linux is supposed to be more performant than other OS's when it comes to file transferring, but lately it's less.  And that damn Plex issue - I've put the mover on minimum everything, but we shouldn't have to - there's clearly some kernel I/O scheduling issue.

No I meant what I wrote. My normal parity check, which runs monthly, hasn't been impacted in any way. The rebuild though.. major impact.

I've even had trouble during this rebuild playing media. We're talking even having disruptions when playing a flac file over my Sonos system at home.

And considering the speed it's going at, it's impacting my performance for almost a week.

[Marshalleq]

OK, I must be reading it wrong.  It sounds like you're saying it's gotten slow, that you're currently getting 50MB/s when you used to get 20, which would mean it's actually faster than what you had before.  I'm confused.... (which wouldn't be the first time)

[Froberg]

1 minute ago, Marshalleq said:

OK, I must be reading it wrong.  It sounds like you're saying it's gotten slow, that you're currently getting 50MB/s when you used to get 20, which would mean it's actually faster than what you had before.  I'm confused.... (which wouldn't be the first time)

Two different things. My REBUILD is going excruciatingly slow, but my regular parity check hasn't seen any performance degradation having been stable at 120-130'ish during the period.

The Rebuild is much, much slower than it was a year ago, when I last replaced a failing drive.

[Marshalleq]

OK.  Remember it will only go as fast as your slowest drive.  And also, if the drives are nearing capacity that is also much slower.  If you have a faulty SATA cable, that would slow down the whole array also.  Those are the main things I can think of off the top of my head.

[Froberg]

Just now, Marshalleq said:

OK.  Remember it will only go as fast as your slowest drive.  And also, if the drives are nearing capacity that is also much slower.  If you have a faulty SATA cable, that would slow down the whole array also.  Those are the main things I can think of off the top of my head.

All the drives are 6-10 TB WD Red's. Once the array is safe, I'll do another Parity check to rule out any cable fault.

[itimpi]

34 minutes ago, Froberg said:

All the drives are 6-10 TB WD Red's. Once the array is safe, I'll do another Parity check to rule out any cable fault.

A parity check will not rule out a cable problem.    Such a problem will only slow down the parity check as the system does retries on an affected drive.   Have you obtained diagnostics (via Tools->Diagnostics) while the slow behaviour is being exhibited as that should help resolve if you have a cable problem.

[Froberg]

Here ya go. It's still working on rebuilding, so it's slow now.

 

 

To be clear though. The system is usable right now. So it's not like the mover issue. I just get intermittent interruptions.

fortytwo-diagnostics-20190806-0254.zip

Edited August 6, 2019 by Froberg

[Marshalleq]

Yikes, that's terrible speed.

[itimpi]

49 minutes ago, Froberg said:

Here ya go. It's still working on rebuilding, so it's slow now.

 

 

To be clear though. The system is usable right now. So it's not like the mover issue. I just get intermittent interruptions.

fortytwo-diagnostics-20190806-0254.zip 336.62 kB · 1 download

The syslog is full of resets on ata12 so that definitely suggests that you have a problem and it explains the speed problems.    Typically it is a cable problem but it can be something else.

[Froberg]

How do I tell which drive is connected to ATA12? I have hot-swap bays, so might just try and move it to a new slot.

[JorgeB]

ATA12 is disk4, and you should really avoid the Marvell 9230 controller on those boards, they are a known problem and tend to drop disks.

[Froberg]

3 minutes ago, johnnie.black said:

ATA12 is disk4, and you should really avoid the Marvell 9230 controller on those boards, they are a known problem and tend to drop disks.

Just now, johnnie.black said:

ATA12 is disk4, and you should really avoid the Marvell 9230 controller on those boards, they are a known problem and tend to drop disks.

Not possible I'm afraid. It's firmware updated (they implied that solved it) and hasn't been an issue though. Running the cache drives on that controller too.

I guess I'll try and move disk 4. Do I wait for the rebuild or just throw it in a new slot after pausing?

[JorgeB]

You should cancel it and connect the disk in a different slot or just replace cables and start over, current issue looks more like a cable problem, but like mentioned if possible don't use that controller.

[Froberg]

The slot seems to have a dead diode anyway, so it's for the best probably.

I'll try. Thanks.

[Froberg]

Server did not enjoy it. I replaced the faulty drive-tray and put it in a different slot.

Now I'm forced in to a disk read check of, it seems, all drives and the drive that was moved is listed as "Disabled". Haven't seen that before.

It's going to take a day just for one drive to read check.. so I'm in for a long haul it seems.

[JorgeB]

You can cancel the read check, just reassign the drive to begin rebuilding.

[Froberg]

I tried that first, but it just kept only showing me the option for read check. Just going to let it run, looks like it's all disks based on read activity and it's going fairly quickly;

[JorgeB]

It will take roughly the same time as a parity check/disk rebuild.

[Froberg]

50 minutes ago, johnnie.black said:

It will take roughly the same time as a parity check/disk rebuild.

Right you are. It was a bit more involved than I had expected. Not sure what I did to trigger this, but maybe the fact that I pulled the failed drive and inserted the new one in the slot the failing one was in, caused some confusion.

 

I had to stop the array, remove the drive from the array, tried to stop the array again but it couldn't unmount the shares for whatever reason, rebooted, stopped the array, assigned the drive again, restarted array - partity check has begun.

 

It's now back on-line and the speeds are looking much improved.

 

Thanks for the assist, and apologies for thread hijacking as the mover issue is still pertinent.

[JorgeB]

27 minutes ago, Froberg said:

and apologies for thread hijacking

Yes, I didn't even notice this wasn't your thread, you should have started a new one.

 

@MaticksI also noticed some i/o issues when using more than one array disk, your problems are likely related to this bug report.

 

 

[Froberg]

1 hour ago, johnnie.black said:

Yes, I didn't even notice this wasn't your thread, you should have started a new one.

 

@MaticksI also noticed some i/o issues when using more than one array disk, your problems are likely related to this bug report.

 

 

Agreed. This appears to be the exact issue. I am experiencing all those same problems. My array-rebuild not withstanding.