Using 10Gb adaptor in PCIe 1x slot (An Unraid 10Gb journey)


KptnKMan

Recommended Posts

1 hour ago, KptnKMan said:

I've never seen that adaptor before, does that convert the physical x1 to a x16 SFF? That's amazing.

The throughput of 6.09 Gbits/sec is a perfect compromise, I can do that.

Did you have any issues with the SFF fitting or does it secure down well?

You're using SFF cards for this to work? I've seen a few different models.

Yes it secures ok. Would be nice if there was a slot and plate at the bottom for the sff card plate to locate into. I can get some pictures if you want.

 

Yes sff

  • Thanks 1
Link to comment

@SimonF I saw some pictures, but I would appreciate a pic if its easy for you.

I saw another item, this one, that has the plate I think you're referring to.

Is it much of an issue without it, I guess its secure with the usual bracket screw?

 

Edit: Looks like the metal bracket wouldn't fit the X1-to-X16 riser anyway.

I've ordered 2 just now, worth it.

Edited by KptnKMan
Link to comment
On 11/2/2020 at 3:25 PM, KptnKMan said:

Trouble is, I've been all over the internet for months looking for details compatibility of running 10Gb adaptors in a PCIe 1x slots.

 

Its possible with an open x1 slot (like on my ASRock board) because the cards have all PCIe 3.0 x4 interfaces - but with speed reduction.

 

grafik.png.d8dbac64dd7a0fd2f321a9241021769f.png

 

See this link for the speeds https://en.wikipedia.org/wiki/PCI_Express

 

grafik.png.5970fef96330f74c7de43f732b819fac.png

 

You can see that a PCIe 3.0 x1 can handle 985MB/s - not really less compared with max. 1250MB/s a 10GBit-NIC can handle.

When i move data to my server over the Asus 10GBit NIC, i reach between 950MB/s and 1100MB/s.

Edited by Zonediver
Link to comment
2 hours ago, KptnKMan said:

@Ford Prefect Yeah this all makes a lot of sense.

Looks like there's even more reason to stretch for the bigger MT switch now, with the HW-offloading for VLANs being something I'd like.

...regarding VLANS: do not confuse L2-HW-Offloading (VLAN filtering/switching, which is a thing all CRS3xx models can do already with ROSv6 in wire speed) with L3-HW-Offloading.

This will basically turn the device into an IP (hence L3) Router (which is what ROS can do in software with v6, but L3-HW offloading will push that to almost wirespeed as well. CRS3xx models have a switch chip and a small CPU. L3-Hardware offloading will enable ROS to do these things on the Switch-Chip, like with L2 and not on the smaller CPU anymore). Not all L3 traffic, but for some features. L3-HW offloading for VLANs will be inter-VLAN communication/IP-routing, hence.

 

Quote

Are the SR tranceivers you're referring to 10Gb capable, as I see the 1Gb tranceivers cheaper when I look around.

Honestly, I'd rather use ethernet if I can as I have spare CAT6/7, so this would be fine for me.

SFP+ are 10G, SFP (non-plus) are 1G.

SR means single mode fiber, which is more robust for a reliable connection than multimode fiber (MM), which is a little cheaper than SR. But actually both will do...just don*t mix the modes ;-) as they are not interchangeable/compatible.

 

If you want to utilize existing copper/CAT cabling, you'll need 10G-T (SFP+ RJ45) transceivers....these are more expensive and use a lot more energy (and hence get really, really *hot*). It is recommended to only use these in every second port only, especially on devices with passive cooling.

The MT S-RJ10 is one...I use a pair over 18m of cat5e wiring (CRS326 to CSS610 as an uplink between ground and first floor) and have a stable 10G connection without a single drop for over a year now.

Edited by Ford Prefect
  • Thanks 1
Link to comment

@Ford Prefect Thanks so much for the pictures, these are perfect for what I was wondering about.

The riser looks like its doing a great job, glad I ordered 2 already.

I didn't realise until seeing the pics that you're using a dual-port 10Gb card.

Are you using those for Dynamic Link Aggregation, Backup Redundancy, Load Balancing or another use?

Since the card bandwidth is limited to 6Gb, you'd get 3Gb out of each if used together?

 

I'm checking ebay.de listings at the moment, and wondering if I should get a single-port or dual-port card?

Listings like this and this have multiple single-port cards, but other listings like this and this have dual-port cards for about the same price as a bundle of single cards.

I found another listing here for a pair of dual-port cards with a couple DAC cables included.

Wondering if that last deal would be an ideal setup for a good start with the MT switch?

 

The 2 included DACs would get me started, but I'd have to look at getting tranceivers if I'm going to expand.

I guess that would give me the most flexibility at this time to expand how I want.

Personally, I'm into redundancy as well so the dual-port cards would allow for that as well if I pick up a few more DACs, I guess?

Link to comment
29 minutes ago, KptnKMan said:

@Ford Prefect Thanks so much for the pictures, these are perfect for what I was wondering about.

Eehrrr...no, this one goes to @SimonF, I think

 

29 minutes ago, KptnKMan said:

I'm checking ebay.de listings at the moment, and wondering if I should get a single-port or dual-port card?

Listings like this and this have multiple single-port cards, but other listings like this and this have dual-port cards for about the same price as a bundle of single cards.

I found another listing here for a pair of dual-port cards with a couple DAC cables included.

the single port connect-x3 are PCIe-x4, while the duals are x8 wide.

Since you only have a x1 port to the CPU, I would not bother with the DUALs.

Should you want to use them some times later, in a real x8 slot, abaility creating traffic with 2x10G (in a bond or not) will still cost a lot of CPU-ressources.

I also found this: link ... comes with DAC (looks like 3m) and low+full profile brackets. I did buy some of these from the same vendor.

Also this is the cheapest, I think: link ...comes with a discount, if you buy more...but only full profile bracket

 

29 minutes ago, KptnKMan said:

Wondering if that last deal would be an ideal setup for a good start with the MT switch?

...they all will work just fine.

 

29 minutes ago, KptnKMan said:

The 2 included DACs would get me started, but I'd have to look at getting tranceivers if I'm going to expand.

I guess that would give me the most flexibility at this time to expand how I want.

Personally, I'm into redundancy as well so the dual-port cards would allow for that as well if I pick up a few more DACs, I guess?

You could use the bond in failover mode, yes....but the MT has only 8 ports, remember ;-)

There are other options though: link 

Link to comment
3 hours ago, KptnKMan said:

Are you using those for Dynamic Link Aggregation

I have 3 servers 40gb between two of them and 10gb from each one to this server. Not really used much was just playing. Maybe will use iscsi, vm migration. Each server has 1 or 2 1gb nics also. Wi!l look to move 10gb into a switch at some point.

Link to comment

Oh Damn, sorry @SimonF wrong shoutout to @Ford Prefect my bad.

 

Thanks to you both for the advice still.

I would like to eventually use the second port for failover, and I don't see any issue with 8 SFP+ ports any time soon.

Currently I'm using a 1Gb unmanaged switch, as I had an old HP Procurve 100Mb switch I used years ago with VLANs, but use the unmanaged now for the 1Gb bandwidth (And it was free and has lots of RJ45 ports).

 

I've been looking at basically a new setup, which is why I haven't purchased anything yet, but just haven't got around to it until recently.

 

I'm planning to use the MT CRS309-1G-8S+IN switch as a 10Gb core for my servers with a 1Gb port for my WAN router and a 1Gb/10Gb uplink to a 1Gb switch for the rest of the network, which I would also need to purchase. I was looking at the CRS326-24G-2S+IN as a good uplink switch with plenty of 1Gb RJ45 ports, but then maybe I should look at a single switch that can do everything... but that sounds more expensive. Alternatively I could purchase any other new/used 1Gb managed switch, doesn't have to be MT branded. Still looking at options there.

 

Anyhow I think maybe having a separate switch would make placement easier for me, as I could leave the 10Gb close to my core servers and continue with the rest of the house hooked into the 1Gb switch.

I think if I used something like the CRS317-1G-16S+RM, then I would also have to purchase 1Gb transceivers for everything, and it seems preferable to maintain my network direct to an ethernet switch. I also want to add more 10Gb later without compromising the 1Gb connectivity.

 

In the end I think I can stretch to the dual-port Mellanox cards, even if I don't use the second port just yet (Especially as the card bandwidth is already limited to 6Gb).

Link to comment

Yes, sounds like a plan.
I do have a RB4011 as a core router and crs326 as core switch. Then css610 as satellite switches ...they are nice, although only SWOS.
Running 10G over 18-20m cat5e with S-RJ10 transceiver.
Smaller 5 port satellite switches, I use dlink dgs1100-05PD....PoE in and can drive another WLAN AP and splitter via PoE as well.

Central 10G hub is the crs309.

Gesendet von meinem SM-G960F mit Tapatalk

  • Thanks 1
Link to comment
  • 2 weeks later...

Well its been an interesting few days ordering parts.

I ordered the PCI-E x1 risers and dual-port 10Gb cards, and they have arrived.

 

20210817_024931.thumb.jpg.0e68795800d530f748a31d84e4307407.jpg

 

I also tried to acquire an CRS309-1G-8S+IN on ebay but the seller refused to ship to Netherlands, from Germany and would not respond to enquiries.

I'm pretty disappointed in that actually, as it would have been perfect.

 

So now I'm looking to purchase a CRS309-1G-8S+IN new with a couple transceivers for other connectivity (I have DACs for servers now).

@Ford Prefect and @SimonF I'm wondering if I can use any SFP 1Gb RJ-45 transceiver or are specific models compatible?

I'm looking at this listing and wondering if it would be compatible or if I should order from fs.com as suggested?

As far as I know, either should be compatible?

Edited by KptnKMan
Link to comment
On 8/17/2021 at 3:12 AM, KptnKMan said:

I also tried to acquire an CRS309-1G-8S+IN on ebay but the seller refused to ship to Netherlands, from Germany and would not respond to enquiries.

I'm pretty disappointed in that actually, as it would have been perfect.

...I do not buy new stuff at the bay, except part originated from china.

For MT in Europe, look at eurodk.de ... at least for Germany, they ship it internally to a distribution centre in Germany and the stuff gets then forwarded via DHL...tracking et al works fine.

The are also listed in price-portals but tend to be more expensive there. When buying directly from them, I think their standard policy for returns is, that you'll have to pay postage and packing yourself/upfront (at least for things you just deem unfit after inspection and return them...not for DOA/damaged stuff).

 

On 8/17/2021 at 3:12 AM, KptnKMan said:

So now I'm looking to purchase a CRS309-1G-8S+IN new with a couple transceivers for other connectivity (I have DACs for servers now).

...see above

On 8/17/2021 at 3:12 AM, KptnKMan said:

@Ford Prefect and @SimonF I'm wondering if I can use any SFP 1Gb RJ-45 transceiver or are specific models compatible?

MT is very much open for every brand, I think...they definitely have no vendor lock.

 

On 8/17/2021 at 3:12 AM, KptnKMan said:

I'm looking at this listing and wondering if it would be compatible or if I should order from fs.com as suggested?

As far as I know, either should be compatible?

 

Yes they should work, but these are just 1G.

I'd not waste a SFP+ port for 1G port on a CRS309...it already has one, btw.

For fan-out of more 1G ports, I'd combine the CRS309 with a CSS610: https://www.eurodk.de/de/products/crs/cloud-smart-switch-610-8g-2s-in (or even use a CRS/CSS326)

Then use a DAC between them

  • Thanks 1
Link to comment
  • 3 weeks later...

Hi, sorry for the delay, had some stuff going on and getting back on track with this now.

 

@Ford Prefect thanks for the link to eurodk.de, that's a great site.

I made up a shopping list some time ago, and want to confirm if it looks about right:

image.thumb.png.51ca053f4d7971e97bc87085b728d10b.png

 

So something critical I'm wondering about, is that I have a distributed setup in my house.

My servers are in one room with the core switch, and I have my desk/office in another room where my current uplink lab switch is.

 

The room my servers are in, there's not much in there at the moment (basically by themselves apart from a couple 1Gb devices), but could I reasonably get away with a couple 1Gb SFP modules? I need at least 1x 1Gb for my WAN router and another for WiFi AP. After that I suppose its more cost-effective to have a CSS610?

I know you said its better to get a CSS610, but is that really a better idea than getting a couple transceivers?

I guess cost-wise it ends up not much more (extra ~40 euros instead of 2 transceivers).

 

If I put the CSS610 on my office desk for lab work, and uplink to it via the multi-mode fiber, will a couple transceivers be better, or should I really just fess up the ~40e for a second CSS610 in the server room with the short DAC?

 

Maybe I'm overthinking it, just get the second CSS610?

I feel like I'm deciding the future of the entire infrastructure right now.

Link to comment
12 minutes ago, KptnKMan said:

Hi, sorry for the delay, had some stuff going on and getting back on track with this now.

 

@Ford Prefect thanks for the link to eurodk.de, that's a great site.

I made up a shopping list some time ago, and want to confirm if it looks about right:

image.thumb.png.51ca053f4d7971e97bc87085b728d10b.png

Technically yes, I think...but will it "fit" into your rooom setup?

I just ordered a pair of these: https://www.amazon.de/gp/product/B091F3RZDF/ref=ppx_yo_dt_b_asin_title_o03_s01?ie=UTF8&th=1 ... they work well with the MT CSS610 and CRS326.

A length of 0.5m of a fiber wire/DAC is really short...mind you that you cannot bend them like the copper counterparts....this length is only good for two ports that quite near.

Also 5m of cable between server room and office room.....mind you, you could still wind up some exxecc-slopes on either end...you can cover 300m ;-)

 

 

 

12 minutes ago, KptnKMan said:

So something critical I'm wondering about, is that I have a distributed setup in my house.

My servers are in one room with the core switch, and I have my desk/office in another room where my current uplink lab switch is.

Do you wnat to replace the core switch with the CRS309 or just add the CRS309 to your setup?

 

12 minutes ago, KptnKMan said:

The room my servers are in, there's not much in there at the moment (basically by themselves apart from a couple 1Gb devices), but could I reasonably get away with a couple 1Gb SFP modules? I need at least 1x 1Gb for my WAN router and another for WiFi AP.

see above.

What exactly are the numbers? The CRS309 has one 1G port already, besides the 8x10G.

You could use this to link to your existing core switch....then add 1G modules as you suggest (if at alle necessary).

 

12 minutes ago, KptnKMan said:

After that I suppose its more cost-effective to have a CSS610?

I know you said its better to get a CSS610, but is that really a better idea than getting a couple transceivers?

I guess cost-wise it ends up not much more (extra ~40 euros instead of 2 transceivers).

If you do not need the extra ports, you don't need the CSS in the server room....you can add it later anyway.

 

12 minutes ago, KptnKMan said:

If I put the CSS610 on my office desk for lab work, and uplink to it via the multi-mode fiber, will a couple transceivers be better, or should I really just fess up the ~40e for a second CSS610 in the server room with the short DAC?

see above...I am more worried about the 5m fiber ... rather go for 10m...for anything below real 7m, you could use a DAC as well.

 

12 minutes ago, KptnKMan said:

Maybe I'm overthinking it, just get the second CSS610?

I feel like I'm deciding the future of the entire infrastructure right now.

Welcome to the club. ;-)

What I do in such cases is, draw a diagramm and make one for each variant. -> https://app.diagrams.net/

My opinion is, that an extra CSS610 will proof more flexible in the future. 

 

  • Thanks 1
Link to comment

@Ford Prefect Sorry, reading back, I think I may have fudged my explanation. My bad, I really appreciate you spending your time to respond to me about this.

I'll try again. 😅

 

So, I'm pretty set on having the CRS309 as my core switch, in the server room, which my servers would DAC directly into. I have 2x 3m DACs with the cards I got already, so I'm super happy about that. 😁

For expansion or other devices (WAN router, WiFi AP, anything else) in the server room, I think could either get a couple 1Gb RJ45 transceivers or short DAC to a CSS610? That is sounding more like a better option, the more I think about it.

 

For the office/desk I'm thinking of another CSS610 with a 10Gb uplink to the core CRS309, which would definitely satisfy that requirement.

Currently I have an unmanaged netgear 8-port switch on my desk so it would be a major upgrade there also.

 

I guess the question becomes how do I uplink from the office/desk to the server room.

 

34 minutes ago, Ford Prefect said:

Technically yes, I think...but will it "fit" into your rooom setup?

I just ordered a pair of these: https://www.amazon.de/gp/product/B091F3RZDF/ref=ppx_yo_dt_b_asin_title_o03_s01?ie=UTF8&th=1 ... they work well with the MT CSS610 and CRS326.

This is a very nice find, I could see getting a 7m DAC for 26euros being perfect for the uplink. 😁

 

36 minutes ago, Ford Prefect said:

A length of 0.5m of a fiber wire/DAC is really short...mind you that you cannot bend them like the copper counterparts....this length is only good for two ports that quite near.

Also 5m of cable between server room and office room.....mind you, you could still wind up some exxecc-slopes on either end...you can cover 300m ;-)

Yeah the 0.5m would be because they would be right next to each other.

You've got a good point though, a 1m or 2m DAC would be a better idea in case I need to move them in the room (Likely).

 

40 minutes ago, Ford Prefect said:

Do you wnat to replace the core switch with the CRS309 or just add the CRS309 to your setup?

Yes, my idea is that the CRS309 would be replace and be my new core switch, and would spoke out to other places in the house.

 

41 minutes ago, Ford Prefect said:

What exactly are the numbers? The CRS309 has one 1G port already, besides the 8x10G.

You could use this to link to your existing core switch....then add 1G modules as you suggest (if at alle necessary).

Doing a quick review now, I have a few things hooked into my current core server room switch:

-both unraid servers (direct, currently RJ45)

-office/desk (uplink via 1Gb RJ45)

-home automation cabinet (uplink via 1Gb RJ45)

-WiFi AP (direct 1Gb RJ45)

-pfSense WAN router (uplink via 1Gb RJ45)

 

I didn't mention before, but the WAN router is on the other side of the house, in the living room where the outside line comes in.

That uplinks via an unmanaged switch, via a very long ~30m CAT5 ethernet cable that I consider "dirty" WAN.

Xbox is on that switch also.

I'm going to rerun at some point, as 2x CAT6/7 ethernet, so that the WAN router has a dedicated ethernet and I can move the WAN router to the server room.

Or I could replace the unmanaged switch there with a CSS610 or something, and I can VLAN it.

Not sure yet.

 

1 hour ago, Ford Prefect said:

If you do not need the extra ports, you don't need the CSS in the server room....you can add it later anyway.

Yeah, looking at my quick review above, I might well need the extra ports of the CSS610 after all. 🤔

 

1 hour ago, Ford Prefect said:

I am more worried about the 5m fiber ... rather go for 10m...for anything below real 7m, you could use a DAC as well.

The 7m DAC might be the best option for me here, for 26 euros it seems simple and ideal. a 24 euro jump to 10m though... not sure if I need the extra length.

I use my primary unraid as my GPU passthrough desktop and have some 7m/5m cables run between the rooms already. Works pretty well. 😁

 

1 hour ago, Ford Prefect said:

Welcome to the club. ;-)

What I do in such cases is, draw a diagramm and make one for each variant. -> https://app.diagrams.net/

My opinion is, that an extra CSS610 will proof more flexible in the future.

I should really do that, draw it out. Gonna have a look at that today.

As it is right now, I think the CRS309 core and 2x CSS601 uplink will work pretty well going forward.

Just need a 1/2/3m + 7m DAC, and I should be good to go.

Link to comment
1 hour ago, KptnKMan said:

Doing a quick review now, I have a few things hooked into my current core server room switch:

-both unraid servers (direct, currently RJ45)

-office/desk (uplink via 1Gb RJ45)

-home automation cabinet (uplink via 1Gb RJ45)

-WiFi AP (direct 1Gb RJ45)

-pfSense WAN router (uplink via 1Gb RJ45)

 

I didn't mention before, but the WAN router is on the other side of the house, in the living room where the outside line comes in.

That uplinks via an unmanaged switch, via a very long ~30m CAT5 ethernet cable that I consider "dirty" WAN.

Xbox is on that switch also.

I'm going to rerun at some point, as 2x CAT6/7 ethernet, so that the WAN router has a dedicated ethernet and I can move the WAN router to the server room.

Or I could replace the unmanaged switch there with a CSS610 or something, and I can VLAN it.

Or go 10G there as well...30m is a bit tight over cat5e...you could try and maybe still end up with a 2.5G or even 5G link.

When you move to cat6// do use these: https://www.fs.com/de/products/87588.html ...they're specified for 80m...technology moves on ;-)

Not sure if these offer fallback to 2.5G/5G or are only 1/10G.

 

1 hour ago, KptnKMan said:

Not sure yet.

 

Yeah, looking at my quick review above, I might well need the extra ports of the CSS610 after all. 🤔

 

The 7m DAC might be the best option for me here, for 26 euros it seems simple and ideal. a 24 euro jump to 10m though... not sure if I need the extra length.

I use my primary unraid as my GPU passthrough desktop and have some 7m/5m cables run between the rooms already. Works pretty well. 😁

10m a bit beyond standard...I would not dare to go above 7m with these.

 

1 hour ago, KptnKMan said:

 

I should really do that, draw it out. Gonna have a look at that today.

As it is right now, I think the CRS309 core and 2x CSS601 uplink will work pretty well going forward.

Just need a 1/2/3m + 7m DAC, and I should be good to go.

....looks like a plan.

Good luck!

  • Thanks 1
Link to comment

Thanks I really appreciate you helping me out with this, and of course everyone else in this thread.

It's been a long time figuring this out, getting feedback, deciding the setup. 😅

 

I just ordered the eurodk kit.

Once I get confirmation, I'll get the amazon order done.

 

Also, yeah everything shows in stock for me.

Not sure why it's not showing up in the screenshot as such.

Edited by KptnKMan
Link to comment

Looks like I'm having fun today:

image.png.97e3ec1eef4484c778b8aea06d278d5a.png

 

And some current iperf3 results between servers:

Quote

root@unraid1:~# iperf3 -c 192.168.178.112
Connecting to host 192.168.178.112, port 5201
[  5] local 192.168.178.111 port 53958 connected to 192.168.178.112 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   110 MBytes   922 Mbits/sec    0    238 KBytes       
[  5]   1.00-2.00   sec   109 MBytes   916 Mbits/sec    0    240 KBytes       
[  5]   2.00-3.00   sec   110 MBytes   920 Mbits/sec    0    243 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   927 Mbits/sec    0    240 KBytes       
[  5]   4.00-5.00   sec   108 MBytes   906 Mbits/sec    0    235 KBytes       
[  5]   5.00-6.00   sec   108 MBytes   908 Mbits/sec    0    238 KBytes       
[  5]   6.00-7.00   sec   109 MBytes   912 Mbits/sec    0    240 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   943 Mbits/sec    0    238 KBytes       
[  5]   8.00-9.00   sec   109 MBytes   916 Mbits/sec    0    238 KBytes       
[  5]   9.00-10.00  sec   110 MBytes   924 Mbits/sec    0   5.66 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.07 GBytes   919 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.07 GBytes   917 Mbits/sec                  receiver

root@unraid2:~# iperf3 -c 192.168.178.111
Connecting to host 192.168.178.111, port 5201
[  5] local 192.168.178.112 port 41160 connected to 192.168.178.111 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   954 Mbits/sec    0    240 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   942 Mbits/sec    0    243 KBytes       
[  5]   2.00-3.00   sec   113 MBytes   945 Mbits/sec    0    238 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   943 Mbits/sec    0    240 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   934 Mbits/sec    0    240 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec    0    240 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   943 Mbits/sec    0    238 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   943 Mbits/sec    0    240 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   943 Mbits/sec    0    240 KBytes       
[  5]   9.00-10.00  sec   111 MBytes   934 Mbits/sec    0   5.66 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

 

  • Like 1
  • Thanks 1
Link to comment
1 minute ago, Ford Prefect said:

Xmas time ;-)

Thanks, I've got everything setup... I even managed to move some hardware around, and I have both the Mellanox NICs in the 2nd PCIe x16 slot.

I'll need to move them at some point, but for now I'll ideally get full speed out of them, and glad I have the x1-to-x16 adaptors. 😄

 

4 minutes ago, Ford Prefect said:

...that is pretty good...are these with the NICs in the x1-PCIe Slots already 🤩

This is from the old 1Gb cards, but one is the onboard 1Gb and an Intel 2x 1Gb card.

I had them bonded but the performance was still lacklustre from what they should be.

I should be getting about 3Gb/s out of the bonded link... but the 10Gb card should do better.

Link to comment
8 minutes ago, KptnKMan said:

This is from the old 1Gb cards, but one is the onboard 1Gb and an Intel 2x 1Gb card.

🤣Sorry, I actually did not wear my Glasses

Edit: this is, what it looks like with my 10G Ports active:

 

image.png

 

8 minutes ago, KptnKMan said:

I had them bonded but the performance was still lacklustre from what they should be.

I should be getting about 3Gb/s out of the bonded link... but the 10Gb card should do better.

Redo the test on the client side by adding "-P 4" .... What Switch is in Between? As the MAC of the bond is the same, it should ackowledge the different ports, in addition to the MAC...but on my MT CRS326, this only works when selecting L2+L3 as extra bonding parameter.

I get 3.5Gbps with a Bond on my i350-T4 in unraid.

Edited by Ford Prefect
Link to comment
1 minute ago, Ford Prefect said:

Redo the test on the client side by adding "-P 4" .... What Switch is in Between? As the MAC of the bond is the same, it should ackowledge the different ports, in addition to the MAC...but on my MT CRS326, this only works when selecting L2+L3 as extra bonding parameter.

I get 3.5Gbps with a Bond on my i350-T4 in unraid.

It was on the old unmanaged Netgear switch I was using until this upgrade.

 

I just finished getting everything installed and setting up the links, and looks like the core CRS309+CSS610 have booted up ok and pushing data.

 

Before I get into setting them up, I seem to have run into a strange issue with the MACs of the interfaces.

The bonding configuration seems to think there are duplicate MAC adresses:

image.png.a810007815d52750fa1ad7a6719baca8.png

 

eth0 is the oboard 1Gb, and eth1/2/3 seem to be the Mellanox dual-port card.

I think I need to delete one of the interfaces, reading up on why this might have happened.

 

If I try to change something in the GUI I get an error:

image.png.ffa1eaf4c44fd0ac89536ddec8a528f7.png

 

🤔

Link to comment

I think I may need to open a new thread for this issue.

I'm trying to setup acive-backup on bond0, so that the (eth1) Mellanox 10Gb port1 is primary, and the (eth0) onboard 1Gb is backup.

 

The duplicate MAC error seems to be blocking it.

From what I'm reading I can't seem to query the bond status.

Edited by KptnKMan
Link to comment
  • KptnKMan changed the title to Using 10Gb adaptor in PCIe 1x slot (An Unraid 10Gb journey)

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.