steve1977 Posted September 2, 2019 Share Posted September 2, 2019 I am not clear on how the number of mobo/cpu lanes impact performance, but hope to find some help from the community. My setup: X299-A with i7 7800X. I have two M.2 disks on board. A M1015 raid controller in first PCI slot and a Nvidia 1060 TI in the second. I am now thinking to add a Asus Hyper M.2 X16 Card V2 to the third PCI slot to add yet anothr four M.2 disks. I read different messages about the feasibility and potential limitations due to the number of lanes. But didn't understand anything. maybe someone can shed some light in it and provide me with some advice? Thanks a lot for your help! Quote Link to comment
tjb_altf4 Posted September 4, 2019 Share Posted September 4, 2019 Acording to intel's spec, that cpu only has 28 pcie lanes note: some of the higher tier cpus will unlock more lanes (44) on X299. https://ark.intel.com/content/www/us/en/ark/products/123589/intel-core-i7-7800x-x-series-processor-8-25m-cache-up-to-4-00-ghz.html You're probably near the limit and most certainly won't have another 16 lanes up your sleeve (probably only 4 spare). Quote Link to comment
steve1977 Posted September 4, 2019 Author Share Posted September 4, 2019 Thanks. How did you come up with the number of 4 spare lanes? Quote Link to comment
JorgeB Posted September 4, 2019 Share Posted September 4, 2019 You just need to look at the board's specs page: Quote 28-Lane CPU- 3 x PCIe 3.0/2.0 x16 (x16, x16/x8, x16/x8/x1) *4 2 x PCIe 3.0 x4 (max at x4 mode) *1 1 x PCIe 3.0/2.0 x1 *3 1. PCIEX4_1 max. at x1 mode, PCIEX4_2 max. at x4 mode 2. SATA6G_5/6/7/8 share bandwidth with PCIEX4_2. 3. PCIEX1_1 shares bandwidth with Front USB 3.1 GEN 2 4. PCIEX16_3 share bandwidth with PCIEX4_1 when using x28 or x16 lane CPU So third x16 slot will run at x1 (if PCIe x4 slot1 is empty) Quote Link to comment
steve1977 Posted September 5, 2019 Author Share Posted September 5, 2019 Thanks. Still a bit cryptic to me Assume the slots are counted started from the CPU? If so, my raid card is currently in the first, my nvidia in the second, and the third one is empty. In order to use the hyper card with 4 M.2 disks, I'd need to plug it into the first slot? If so, could the Nvidia or the raid card function in the x1 slot? Even if I upgrade the CPU, I could not use the hyper card in the third slot (only x8)? Can I change what slot uses how many lanes? I.e., from 16/16/8 to 16/8/16? Thanks!!! Quote Link to comment
JorgeB Posted September 5, 2019 Share Posted September 5, 2019 2 minutes ago, steve1977 said: Assume the slots are counted started from the CPU? No need to assume anything, just check the manual, but usually they are counted from the top down. 10 minutes ago, steve1977 said: If so, could the Nvidia or the raid card function in the x1 slot? Depends, some HBAs don't work on a x1 slot, don't know about GPUs, but even if it does bandwidth will be very reduced. 12 minutes ago, steve1977 said: Even if I upgrade the CPU, I could not use the hyper card in the third slot (only x8)? Correct, you also need to check that slot1 supports pcie bifurcation, or it will only detect one device. 12 minutes ago, steve1977 said: Can I change what slot uses how many lanes? I.e., from 16/16/8 to 16/8/16? No Quote Link to comment
steve1977 Posted September 6, 2019 Author Share Posted September 6, 2019 I have now moved my HBA to slot 3. GPU stays in slot 2. Slot 1 is empty for now. It seems everything is working. Does this mean I am now running on x16/x8 or on x8/x1. Do HBA and GPU benefit from x16 over x8? I did some googling and it seems the difference is negligible for GPU. Didn't find any info for HBAs. I am considering to upgrade the CPU to get all three cards working. I could upgrade to a CPU with 44 lanes or wait a few month as Cascade Lake seems to feature 48 lanes? With 48 lanes, I'd be able to run all 3 cards as x16? Quote Link to comment
JorgeB Posted September 6, 2019 Share Posted September 6, 2019 8 minutes ago, steve1977 said: Does this mean I am now running on x16/x8 or on x8/x1. You can check with: lspci -vv 8 minutes ago, steve1977 said: Didn't find any info for HBAs. Your HBA is x8. 1 Quote Link to comment
steve1977 Posted September 7, 2019 Author Share Posted September 7, 2019 Just run it, but not fully clear to me. Please see results https://pastebin.com/s8N6q3As Quote Link to comment
JorgeB Posted September 7, 2019 Share Posted September 7, 2019 It's running x8/x1 GPU LnkSta: Speed 8GT/s (ok), Width x8 (downgraded) HBA LnkSta: Speed 5GT/s (ok), Width x1 (downgraded) Quote Link to comment
JorgeB Posted September 7, 2019 Share Posted September 7, 2019 BTW, x8 is fine for the GPU, but x1 will severely bottleneck the HBA. Quote Link to comment
steve1977 Posted September 7, 2019 Author Share Posted September 7, 2019 Got it, thanks! On a positive note if this setup works fine, I could add the hyper M.2 card in the first slot and have the full x16 (i.e., four NVME)? Or an issue as the on-board NVME may also require 4 lanes to add up to 29 instead of the 28 available lanes? When you say severely bottleneck, what are the implications? Is the array and parity impacted? Would it be unstable or just a bit slower? Lastly, I saw the new CPUs for X299 will feature 48 of 44/28 lanes. Any thoughts whether this would require a new mobo as well? I am thinking to upgrade and wonder whether to wait for Cascade Lake X with 48 lanes or just go with the current 44 lanes CPUs. Quote Link to comment
JorgeB Posted September 7, 2019 Share Posted September 7, 2019 35 minutes ago, steve1977 said: On a positive note if this setup works fine, I could add the hyper M.2 card in the first slot and have the full x16 (i.e., four NVME)? Yes, if your board supports PCIe bifurcation for slot1, check the manual. 35 minutes ago, steve1977 said: When you say severely bottleneck, what are the implications? Mostly parity checks and disk rebuild, also all writes if you use turbo write. 36 minutes ago, steve1977 said: Any thoughts whether this would require a new mobo as well? Sorry, don't known, never needed to find that out, but it shouldn't be difficult to find with google. Quote Link to comment
steve1977 Posted September 7, 2019 Author Share Posted September 7, 2019 Thanks again! I have turbowrite (Tunable (md_write_method)) set to auto. It copies at 150mb/s, which is the same speed that I used to have before and also the same I had during parity rebuilds (when HBA was in slot #1). So, it indicates that nothing changed? This does not seem like expected behavior? Quote Link to comment
JorgeB Posted September 7, 2019 Share Posted September 7, 2019 3 minutes ago, steve1977 said: I have turbowrite (Tunable (md_write_method)) set to auto That is the same as disable, and it only affects copy speed to the array, not to cache. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.