Kioxia 7.68TB PCI-E - too good to be true?

[eribob]

Hi all! 

I found these listings on ebay (several similar ones are there, all from china): https://www.ebay.com/itm/374341484174?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=OxLNCOGvR0q&sssrc=2047675&ssuid=6Vb57j3CRuK&widget_ver=artemis&media=COPY

 

They sell Kioxia NVME U.2 drives 7.68TB that are PCIE gen 4 capable as far as I understand. Great read and write speeds and extremely durable. Price around 500USD. 

 

I am hooked and would like to put 2 of them in my unraid server but I do not want to fall into some loophole that I did not think about, since it is still a lot of money. So I have some questions for you pros! 

 

1. Have anyone bought these from the chinese sellers? Are they scamming you or are they for real? The price seems very (too?) good and they claim that the drives are brand new... At the same time the sellers are top rated with good reviews.

2. A PCI-E adapter: I have an Asrock Taichi X570 motherboard with a ryzen 3950x processor, so not a lot of PCIE lanes but enough for 2 more drives. I think the board should be capable of PCIE bifurcation (hard to confirm online though) and therefore wanted to buy something like this: https://www.ebay.com/itm/304837578258?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=u0r4ab5cr6-&sssrc=2047675&ssuid=6Vb57j3CRuK&widget_ver=artemis&media=COPY with proper cables. The motherboard has 3 slots that are x16 size but they are running at x8 x8 x4 speeds unless you only populate one of them. I will have a GPU (GTX1070Ti) as well in the x8 first slot and a network card in the x4 slot. An x8 slot should still have enough bandwidth to run 2 drives at full speed, and maybe 4 drives at half speed if I buy more later on? Or how would that work with the bifurcation?

- The alternative is to buy a card with a PCI-E switch on it but they are more expensive and I have not found them in PCI-E 4.0. 

3. What about airflow? Will the drives or the adapter become very hot? I have a normal non-server chassi with quiet fans... 

4. Software configuration: I was thinking to use zfs and run them in the equivalent of raid 0. Backups will be regularly made to the hard drive array that has parity. They have awsome endurance and are expensive so I do not want to waste space by running them in raid 1. 

5. Networking: To utilise the drives I want to add a 10GbE NIC to the final x4 slot. I have CAT6a wired at home so I want to use RJ45. Except for the price (especially on switches... man they are expensive!), are there disadvantages of using RJ45 over SFP+ for 10GbE?

6. CPU and RAM requirements: I have 128Gb of RAM and the CPU have high single thread performance. I was planning not to use zfs cache in the RAM since the drives are so fast anyway, does that make sense? I need the RAM for VM:s including my workstation/gaming VM that I use regularly with the GPU passed through.

 

Looking forward to responses!  

[domrockt]

  

On 7/15/2023 at 10:38 PM, eribob said:

They sell Kioxia NVME U.2 drives 7.68TB that are PCIE gen 4 capable as far as I understand. Great read and write speeds and extremely durable. Price around 500USD. 

 

 

 

i bought 3 of them 2 are here 1 is on its way.

 

i use an chinese 16x to 8x4x4x adapter to m2 to u.2 cables pretty jank but it works just finde

 

i bought mine used they have around 1PB write&read so for my usage as good as new

Edited November 1, 2023 by domrockt

[eribob]

On 11/1/2023 at 6:13 PM, domrockt said:

  

 

 

i bought 3 of them 2 are here 1 is on its way.

 

i use an chinese 16x to 8x4x4x adapter to m2 to u.2 cables pretty jank but it works just finde

 

i bought mine used they have around 1PB write&read so for my usage as good as new

cool! 
can you link the adapter you used? 
 

raid0 like a boss :)
 

thanks! 
/Erik

[domrockt]

39 minutes ago, eribob said:

can you link the adapter you used? 

 

https://de.aliexpress.com/item/1005004837842768.html?spm=a2g0o.order_list.order_list_main.10.4a3b5c5fyI0HHK&gatewayAdapt=glo2deu <---- this works fine

 

https://www.amazon.de/dp/B09DCXYMCZ?psc=1&ref=ppx_yo2ov_dt_b_product_details <----- this for the m.2 next to the GPU 

 

https://www.amazon.de/dp/B0BC1NH63L?ref=ppx_yo2ov_dt_b_product_details&th=1 2x this 

 

https://www.ebay.de/itm/175779281579 2x this cable 

 

39 minutes ago, eribob said:

raid0 like a boss

 

just games and temporary data 😀 everything on here is backed up on the ZFS HDD array raidz1

 

the third nvme arrived, i bought them from this seller 

https://www.ebay.de/itm/285142558990?mkcid=16&mkevt=1&mkrid=707-127634-2357-0&ssspo=si3ztlnrrby&sssrc=4429486&ssuid=4iNN0sYtT6G&var=&widget_ver=artemis&media=COPY

 

they send them via FedEx, ridiculus fast Shipping 

 

it can take up to 5 Days until they are on the way but then oh Boy i Never had anything this fast from China.

 

 

Edited November 12, 2023 by domrockt

[eribob]

Thanks alot! Especially for the ebay seller tip.

 

11 hours ago, domrockt said:

just games and temporary data 😀 everything on here is backed up on the ZFS HDD array raidz1

That is how I was planning to do it as well. 

[eribob]

On 11/12/2023 at 9:39 AM, domrockt said:

https://www.ebay.de/itm/175779281579 2x this cable 

Hi again! 

Finally got around to buying an M.2 drive! I ended up with an intel p4510 8TB drive instead. It is PCI-e 3.0 but I figure that is enough and the IOPS are great. 

Anyway, it works fine, but I am getting slower read/write speeds than expected. I tested writing a large file from the drive to a RAM disk and got around 1.2GB/s (expected around 3GB/s). I am wondering if the cable is the issue? 

 

I bought this cable: https://www.amazon.se/dp/B097BDG3TX/ref=pe_24982401_503747021_TE_SCE_dp_1 

It says SAS 12G or internal NVMe, could it be that it is limited to 1.2GB/s? Perhaps 100cm is too long to allow for max speeds? In that case I will buy the same cable you bought and try again. 

 

The PCI-e card I bought is this one: https://www.amazon.se/dp/B0B6CJ889T/ref=pe_24982401_503747021_TE_SCE_dp_1 - could it be the problem? However, it should just be a dumb link from the PCI-e slot to SFF-8643 ports so perhaps it should be fine? 

 

/Erik

[domrockt]

It is the wrong cable you need to Watch out for PCIe 4x specs with your cable you could use the NVME on an SAS Controller at 12gbits Speed.

 

https://amzn.eu/d/iX79p4e This are the correct ones 

 

Sure the ones from the eBay listing are correct to.

Edited February 18 by domrockt

[eribob]

24 minutes ago, domrockt said:

It is the wrong cable you need to Watch out for PCIe 4x specs with your cable you could use the NVME on an SAS Controller at 12gbits Speed.

 

https://amzn.eu/d/iX79p4e This are the correct ones 

 

Sure the ones from the eBay listing are correct to.

thanks, did not read carefully before buying. Placed an order for that cable now.