Ghosts in the machine type question....

[opentoe]

I threw together an all SSD unraid box. Just using 1 parity drive and 3 500GB SSD drives. I put a few movies on that system and did some read tests which came back at a very good rate of 30-50MB/sec with my media players built in read test. Now, when I do the same read test with my all mechanical hard drive unraid system I get 12MB/sec read test. And that's using the same Ethernet line. I know some people will say the decrease is because it is an all mechanical drive system BUT it is just a read test only. There isn't anything being written to the array or the parity drive. I even threw a couple movies on my cache drive ( don't use it as a cache drive ) which is an SSD drive and the speed tests were still 12MB/sec. Something, somewhere is really slowing down my system and I have no idea where to start with this one. I can see if I was doing write tests and other things were happening at the same time, but these are both controlled tests. I can barely watch any high bitrate movies anymore with my main unraid server, which is really a pain in the ass since it is holding 66 terabytes of data. I have 2 parity drives on that box and 15 mechanical drives and one 500GB SSD cache drive for misc use. Can anyone recommend a single place to start with this one? I have included the built in diagnostics log just in case someone wants to take a peek. I really can't afford to put together an entire box that can hold 66 TB of data or even find an available place to offload that data so I can reformat all my drives, etc. I did notice this I believe when my level of capacity started to approach over %75. Now it is %81. I really hope it is something really obvious, but it seems unraid is degrading speeds the longer I use it and the more capacity I load on it. Thanks.

sun-diagnostics-20190310-1227.zip

[JonathanM]

1 hour ago, opentoe said:

Can anyone recommend a single place to start with this one?

Try temporarily adding a different gigabit ethernet card and disable the one currently in use.

[opentoe]

37 minutes ago, jonathanm said:

Try temporarily adding a different gigabit ethernet card and disable the one currently in use.

I will do that. Since all my boxes have them built on I'll need to order one. I'll grab a popular Intel model. They shouldn't be that expensive.

Thanks, let you know what happens from there.

[opentoe]

5 hours ago, opentoe said:

I will do that. Since all my boxes have them built on I'll need to order one. I'll grab a popular Intel model. They shouldn't be that expensive.

Thanks, let you know what happens from there.

Well, this is a revelation. Whatever was done from V6.6.6 to V6.6.7 fixed the issue. Before I did anything I backed up my flash stick, updated the OS via the web gui, rebooted the server and tried a read test. Now I get 30+ MB/sec read tests on my main unraid system. Version 6.6.6 never went over 12MB/sec. Maybe SMB versions, tweaks or something that's just not mentioned? So right now I'm not fooling around with anything.

[JorgeB]

9 hours ago, opentoe said:

Maybe SMB versions, tweaks or something that's just not mentioned?

No, the only change is the docker, you likely have some network related issues, 12MB/s is 100Mbit Lan.

[opentoe]

12 hours ago, johnnie.black said:

No, the only change is the docker, you likely have some network related issues, 12MB/s is 100Mbit Lan.

My entire network has been 1Gbit for a couple years now. Switches, hubs, cards, haven't used anything in the 100Mbit realm in a very long time. Basically my unraid box plugs into a consumer Netgear 1GB switch which that plugs into my Asus Router. That's all been the same for probably years. Updated to 6.6.7 and for the heck of it tried a read test before I started tearing things apart and was getting 30+ MB/sec tests which is what it is supposed to be. Right now I'm keeping hands off for a while, to see how long this bliss lasts.

[JorgeB]

9 hours ago, opentoe said:

My entire network has been 1Gbit for a couple years now.

Yes, but likely a connection somewhere downgraded to 100Mbits, then fixed on reboot/power cycle.

[opentoe]

14 hours ago, johnnie.black said:

Yes, but likely a connection somewhere downgraded to 100Mbits, then fixed on reboot/power cycle.

I wish I can say that was it. That's ALWAYS the first thing I do. Power cycle everything, even the hubs, switches, everything that's in the loop. I have two identical DUNE media players. One in the living room and one in the bed room. The one in the living room has a straight line going right from my Asus router's 1Gbit ports right to the media player. Been like that since I installed the router. Now that's flowing at the speeds it should I'm happy about that. Now, just tested the one in my bed room. That still stuck at 12MB/sec. But it is going through a Netgear switch. In a little bit I will temporarily run an Ethernet line on the floor directly from the Asus router right into the media player and see the results there. If it picks up speed to its normal 30+ MB/sec then it could be that maybe the media players just do not like Netgear switches. Technically shouldn't make a difference at all since the workstations I have running are all going through switches and I get a constant 950 Mbit speed. Both media players are using the identical firmware.

 

BTW, is there a plugin that shows the current bandwidth for each connection in real time? I do have the Nerd tools installed, but just have a couple utilities installed. Maybe something like that is in there. Preferably a nice looking GUI one. So many plugins. Not complaining, awesome to have so many!

 

Edited March 12, 2019 by opentoe

[opentoe]

Well, back to 100Mbit. Checked the line speed at the switch and that indicates 1Gbit. I was in the middle watching a movie and noticed it started to pause and lag on me. Did a read test and BAM, 12MB/sec max again. Literally nothing was changed. Right before I started watching the movie I confirmed it was reading properly and did a read speed test and it worked fine. 45 minutes into watching the movie something happened and it dropped right back down to 12MB/sec. So I copied the movie file onto my small test box that's hooked directly into the same switch and that's getting 30+ MB/sec no problem. So it literally is boiling down to the main unraid server itself. I did a diagnostic and attached it, I've found nothing. It is pretty much a plain system. So I got normal speeds for 3 days and the only thing I changed was changing that MD WRITE method to "reconstruct". I doubt that has any bearing on the situation since I'm just doing read tests here. So odd. My little three drive test system works fine, but my main system does not. I really wish I can off load 53TB of data, format the drives and start clean. So, the line coming right from the unraid server is indicating 1Gbit from the switch lights.  Here it is almost 4am and the ONLY thing I have not done is used another NIC. My main unraid system has three of them installed....but if it worked fine for 3 days then why all of a sudden it just stops? ARGHHHH, pulling my hair out. I really do think there is some kind of corruption/bug in my main unraid server. Having to perform a speed test right before kicking off a movie and coming back with great results then 45 minutes later, all by itself, back down to 12MB/sec. I'm literally lost here. I've attached my diags BEFORE when things were working and also after a reboot when things degraded back down the old world. Maybe other eyes reviewing the syslog can pick something up?

 

LFKJERIJFOIJFIORW:

ANd while I was writing this I just tried one more time for a read test on that movie file and what would you know, right back to 30+ MB/sec. That's with power cycling nothing and make not 1 single change. I just walked into my living room, performed the read test, expecting 12MB/sec and there it was again 30+MB/sec read. What the hell. I can't take much more of this right now. Too many hours just trying to find out the issue and I have to wake  up in 3 hours and get ready for work. It's just explainable right now. 

 

sun-syslog-20190313-0227_BEFORE THE REBOOT.zip

sun-syslog-20190313-0312_AFTER REBOOT.zip