klamath

So I would let it run NON CORRECTING parity check, I was having issues with my unraid server a few months ago, same issues with freezing etc, turns out let a correcting parity check run and it fucked up a ton of files, 1200 "errors" it found. blake2 confirmed unraid corrupted the files. Not sure why unraid had a bug up its ass but i avoided 6.4+ since then. Tim

Can never hurt to have a backup.

LAG is the shittiest thing ever, TCP cannot be stripped over multiple lines unless your doing round robin and if you have a out of order packet your shit is toast, RR isnt LACP compliant. TLDR, LAG is total shit for two node communication. You can do some stuff with multiple IPs and sharding of connections but for $40 is not worth it! Tim

Id hit that amazon store up, since those cards came from working servers it will assure you the things worked for a while without issue, and you cant really say no to $40. Tim

Just one last hail mary, you might want to disable tso,gso etc from the card, just to rule out any type of offloading. something like this: #ethtool -K eth0 rx off tx off sg off tso off ufo off gso off gro off lro off

The fact it points out nat, mlx4 and a bunch of networking functions points to a spinlock of some kind trying to access the hardware. Seems that you are not the only one having issues: https://www.google.com/search?q=Mellanox+6801-M+crashing&oq=Mellanox+6801-M+crashing&aqs=chrome..69i57.6512j0j7&sourceid=chrome&ie=UTF-8#q=mellanox+6801-m+cable+unplugged Id return those cards and buy the used ones from amazon or intel cards. I get the feeling those might be shitty knock offs or they are cheaply manufactured cards with mellanox chipset thrown on. either way shit seems pretty broke, last ditch id still recommend ubuntu live boot and see if the problem carries over. Tim

Id pick up those cards i linked on amazon, intel cards would work they just cost a pretty penny. Tim

single port, i tired to find more info on your card, seems to be only sold via that newegg link or one seller on amazon. Wondering if these are white label cards or resold tooling to other ODMs: https://www.linkedin.com/pulse/several-popular-models-10gbe-cnanic-hilda-hu

Mine looks to have the same output: C:\WINDOWS\system32>flint -d /dev/mst/mt26448_pci_cr0 query Image type: FS2 FW Version: 2.9.1200 Rom Info: type=PXE version=3.3.400 devid=26448 proto=VPI Device ID: 26448 Description: Port1 Port2 MACs: 0002c952b5be 0002c952b5bf VSD: PSID: MT_0F60110010

The cards are bad or something in your bios is mucking things up, you could try to do a bios reset to factory and live boot ubuntu and try. The cards linked are single spf+ not dual like your cards (im assuming dual because your output shows two mac addrs) Tim

That is what id refer to as fucked might want to try these out, pulled from servers: https://www.amazon.com/Mellanox-Connectx-2-PCI-Epress-Interface-MNPA19-XTR/dp/B016OYD0D4/ref=sr_1_1?ie=UTF8&qid=1499710485&sr=8-1&keywords=mellanox+connectx-2 Tim

Did you disable SR-IOV?

If you live in a populated area there are probably a few stores you can pop into and pick up a new twinax or fibre today, or prime now/prime same day. SHI/Frys/Altex (TX)

I used some old qlogic gbics i had laying around, these *SHOULD* work (i use those in my dlink switch): https://www.amazon.com/gp/product/B00U8Q7946/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 This is suppose to be for mellanox cards: https://www.amazon.com/gp/product/B01D84KUBS/ref=oh_aui_search_detailpage?ie=UTF8&th=1 cable: https://www.amazon.com/gp/product/B00X7TE91M/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

If you disable from the bios it doesnt matter what the card has enabled, you might want to get some fibre gbics and do away with the twinax, id assume it would be cheaper! TIm

I think ryzen support is only hammered out in the RC releases, I had issues with 6.3 releases so i havent been paying attention to the newer stuff, i could be totally wrong. The SR-IOV is only a system bios thing and should be disabled. You can live boot your unraid + windows 10 systems to have a level playing field, install iperf and let it rip for a while. TIm

You can try to copy over the utility program to unraid but you will probably face some missing library issues, you could always live boot ubuntu and test it out that way. Id say 20% you got bad cards, 50% bad cable and the rest a weird config in the bios. If i was you, live boot ubuntu and run a long ass ping test or iperf between machines. Tim

Might want to force a MTU of 1500 just to make sure. Jumbo frames and 10G can have interesting results! Tim

My freenas server uses fibre and has zero issues, my windows 10, unraid servers both have issues on boot sometimes where the link never comes up, disabling the nic or pulling the twinax out and then back in usually resets it. Never had the link drop mid flight like what your describing. Tim

I hear ya, i bought mine used from amazon for $20 a card, got 4 of them in my house all working well besides the twinax acting a fool on startup sometimes, to lazy to replace with fibre. Tim

Possible yes, for an enterprise card that ships as defective the odds are pretty low but are non-zero. If you got a realtek nic or some cheap POS the likelihood of it being defective goes up. Heck i had a defective twinax cable so anything is possible.

That looks right, my only other recommendation is looking into bios features and power savings that could be interfering. Tim

Depending on how you got the cards don't be shocked in the firmware versions dont line up with mellanox proper firmwares, each var has their own "sauce" firmware on it, id recommend just reflashing with proper mellanox firmware.

There are cli tools you can download, a good rubber meets the road post here: https://community.mellanox.com/thread/1858 Tim

Id check the best practices for these cards and make sure your bios is configured correctly, https://community.mellanox.com/docs/DOC-2489 There are others out there, the above is a good example and starting point. TL;DR some bios features can interfere with the card, also make sure your up to date on bios, i had to do some tricky things with my cards since they came pre-flashed with HP bios. Tim