(Solved) need some serious help with read speeds

[Max]

so guys the thing is, i have been using my unraid without any parity drive or cache for past 2 years now, actually 2 years back when i set it up i wasn't getting good write speed to timemachine share so ended up using seprate drive on hackintosh itself so i never really bothered with parity and cache as i was using unraid for only media storage now but finally after 2 years i started to feel that now that my data has grown and i have decent media collection i should get parity drive and cache drive.

---- Here comes the Problem!!! --- from past couple of weeks my read speeds have gone hill from full fat gigabit/sec to like 20-45 megabyte/sec, but still i thought that its maybe because my data has grown now and im using more dockers now too, it could just that my traditional HDD's can't keep up, so maybe everything will be back to normal after installing parity and cache drive as im still waiting for my parity drive to be delivered. (i am getting full fat gigabit/sec write speeds so it was weird but still). But now today all of a sudden my reads speeds have dropped down even further, right now im getting only 2.5 megabyte/sec!!!!!!. 

Now that gotta be something else entirely, i have checked my lan cables, tried diffrent sata ports on my motherboard but still reads are capped to 2.5 mbyte/sec and write are normal 110-112 mbyte/sec.

i have attached my diagnostics, if anyone knows whats wrong here, then please help !!!. i would really appriciate any kind help.

unraid-diagnostics-20190722-1335.zip

[JorgeB]

You're having problems with disk1:

 

Jul 22 18:08:56 Unraid kernel: md: disk1 read error, sector=2286221472
Jul 22 18:08:56 Unraid kernel: md: disk1 read error, sector=2286221480
Jul 22 18:08:56 Unraid kernel: md: disk1 read error, sector=2286221488
Jul 22 18:08:56 Unraid kernel: md: disk1 read error, sector=2286221496

 

Looks more like a connection problem, replace cables.

[Max]

@johnnie.black dude, did someone ever told you how great and awesome you are, just one look and you nailed it. I too at first thought maybe its a loose connection but never thought about trying a different one, as these cables aren't really that old but when you recommended trying a different cable, i took out one from my main rig and replaced it with the cable which was connected to my disk1 and voilla back to full fat gigabit/sec.

Thanks man, i really appreciate it.

[Max]

@johnnie.black 😭looks everything is still not ok. now some are transfering at decent speed and some aren't, i mean its so weird that it can open 4k hdr hevc instantly and i can easily skip ahead without any delay but with some other just 1080p h.264 movies i can easily notice couple of second delay. through netdata i noticed that its transfering data at goods speeds iowait isn't all that bad but when its im only getting like 20megabyte/sec io wait goes to 70-80%.

in this screenshot u can see that on right iowait is hovering around 70-90% thats when its transfering at slowly and on the left side iowait is much better thats when im getting goods speeds.

all testing is done after shutting off all the dockers ecept netdata.

unraid-diagnostics-20190722-2129.zip

Edited July 22, 2019 by Max
adding diagnostics again

[JorgeB]

Nothing visible in the logs, try reading from one or the other disk to see if there's any difference.

[Max]

44 minutes ago, johnnie.black said:

Nothing visible in the logs, try reading from one or the other disk to see if there's any difference.

already tried that now its disk2's read speeds that are slow still write speeds are fine also im still getting iowait spikes according to netdata as soon as i lauch plex so is also running quite slow.

[JorgeB]

Don't know if these can make a difference without parity, but it helped some before with bad read performance, and it won't hurt to try:

 

Settings -> Disk Settings

Tunable (md_num_stripes): 4096
Tunable (md_sync_window): 2048
Tunable (md_sync_thresh): 2000

 

[itimpi]

You might want to consider installing the diskspeed docker and using that to run tests on your disks.   Sometimes you find that disks develop 'slow areas' so and as a result the performance of a drive may widely according to which files you are accessing.   That docker makes it obvious if you have drives that are behaving like that as well as giving information of overall performance of each of the drives you have installed.

[Max]

1 hour ago, johnnie.black said:

Don't know if these can make a difference without parity, but it helped some before with bad read performance, and it won't hurt to try:

 

Settings -> Disk Settings

Tunable (md_num_stripes): 4096
Tunable (md_sync_window): 2048
Tunable (md_sync_thresh): 2000

 

Okay that did did helped quite a bit earlier my read speeds jumping all over between 20-112megabytes/sec, most of the time it would be around 40-45 now with this applied it ranges between 60-100megabytes/sec, mostly around 75-80 megabytes so average speed, minimum speeds are all better but still not as good as my disk1, compared to my disk1 which is giving speeds between 100-112megabytes/sec(i think this has improved it even further and more stable) but it could be that at that when i was checking read speeds, plex was might be reading or writing on disk2 maybe thats why they weren't as good as disk1's. However no one was using plex at that time but still it could be some background process or something, not sure.

58 minutes ago, itimpi said:

You might want to consider installing the diskspeed docker and using that to run tests on your disks.   Sometimes you find that disks develop 'slow areas' so and as a result the performance of a drive may widely according to which files you are accessing.   That docker makes it obvious if you have drives that are behaving like that as well as giving information of overall performance of each of the drives you have installed.

Okay i will try that share the results with u guys, btw we should run that with all our dockers stopped right or doesn't matter??

[itimpi]

8 minutes ago, Max said:

Okay i will try that share the results with u guys, btw we should run that with all our dockers stopped right or doesn't matter??

You want to run it while there is nothing else running that can cause disk activity (dockers, VMs etc) as they could skew the results.  There is a lot to be said for booting in safe mode to run it as that would stop any plugins from running.

[Max]

19 minutes ago, itimpi said:

You want to run it while there is nothing else running that can cause disk activity (dockers, VMs etc) as they could skew the results.  There is a lot to be said for booting in safe mode to run it as that would stop any plugins from running.

Okay then i should try that with unraid os safe mode (no plugins no GUI) right? Thats just unraid's gui, i would still be able to access web gui right?

[itimpi]

1 minute ago, Max said:

Okay then i should try that with unraid os safe mode (no plugins no GUI) right? Thats just unraid's gui, i would still be able to access web gui right?

The GUI will be irrelevant if you are not doing anything with it.   The local GUI is identical to the web based version anyway - just running the browser locally on the Unraid server.  

[Max]

43 minutes ago, itimpi said:

The GUI will be irrelevant if you are not doing anything with it.   The local GUI is identical to the web based version anyway - just running the browser locally on the Unraid server.  

Okay thanks i will try that tonight and report back here.

[Max]

@itimpi @johnnie.black okay these are that results.

graph looks fine to me but im not an expert.

but while the test was running on disk2 is was getting some speedgap error which i didn't get while it was running on disk1. could it bee the problem here ??

[JorgeB]

Looks fine to me.

[Max]

3 minutes ago, johnnie.black said:

Looks fine to me.

what about speed gap thing on disk2, is that too is normal ?

[JorgeB]

3 minutes ago, Max said:

what about speed gap thing on disk2, is that too is normal ?

Not very normal if nothing else was accessing the disk.

[Max]

1 minute ago, johnnie.black said:

Not very normal if nothing else was accessing the disk.

okay just now i tested again by copy pasting some data from disk2 to my main rig's ssd and now this speeds were much better between 80-112 megabytes which is fine i think but that speed gaap ha me worried as no plugin or dockers were ruuning so im pretty sure nothing else was accessing my disk's.

what could caused it, is my disk 2 is dieing or something.

[Max]

are these iowait spikes normal? i get these iowait spikes after launching plex with just plex running.

[Max]

i have been testing my read speeds since yesterday so far so good, everything seems back to normal. thank you for all your help @johnnie.black @itimpi.👍