i not 100% sure on the readings.. i used the iperf3..  and i wanna know is it copying at 10gig network speed or 1gig network speeds. since i bought 10 gig cards and a switch...  but it seems when i copying unraid to unraid my spinning hds only getting 33megs/s  its howing on reading..  and i disabled my onboard nics as i use it for management port to turn on the servers.. but if this is wrong readings for 10gig   what should the values look like?  i noticed issues as it was copying 100 gig unraid back files to my main server.. and it was taking more then an hour  

 

^Ciperf3: interrupt - the server has terminated
root@Rambo:~# iperf3 -c 192.168.0.3
Connecting to host 192.168.0.3, port 5201
[  5] local 192.168.0.23 port 41202 connected to 192.168.0.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.07 GBytes  9.18 Gbits/sec   61    467 KBytes       
[  5]   1.00-2.00   sec  1.07 GBytes  9.17 Gbits/sec   93    478 KBytes       
[  5]   2.00-3.00   sec  1.07 GBytes  9.16 Gbits/sec    4    532 KBytes       
[  5]   3.00-4.00   sec  1.07 GBytes  9.19 Gbits/sec    6    484 KBytes       
[  5]   4.00-5.00   sec  1.07 GBytes  9.16 Gbits/sec   27    478 KBytes       
[  5]   5.00-6.00   sec  1.06 GBytes  9.11 Gbits/sec   10    478 KBytes       
[  5]   6.00-7.00   sec  1.06 GBytes  9.08 Gbits/sec  401    464 KBytes       
[  5]   7.00-8.00   sec  1.06 GBytes  9.10 Gbits/sec   43    509 KBytes       
[  5]   8.00-9.00   sec  1.08 GBytes  9.24 Gbits/sec    0    535 KBytes       
[  5]   9.00-10.00  sec  1.07 GBytes  9.17 Gbits/sec   13    509 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.7 GBytes  9.16 Gbits/sec  658             sender
[  5]   0.00-10.00  sec  10.6 GBytes  9.15 Gbits/sec                  receiver

iperf Done.
root@Rambo:~# iperf3 -c 192.168.0.3
Connecting to host 192.168.0.3, port 5201
[  5] local 192.168.0.23 port 51466 connected to 192.168.0.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.07 GBytes  9.19 Gbits/sec  405    458 KBytes       
[  5]   1.00-2.00   sec  1.06 GBytes  9.11 Gbits/sec    0    526 KBytes       
[  5]   2.00-3.00   sec  1.06 GBytes  9.10 Gbits/sec   82    665 KBytes       
[  5]   3.00-4.00   sec  1.06 GBytes  9.14 Gbits/sec    8    450 KBytes       
[  5]   4.00-5.00   sec  1.07 GBytes  9.16 Gbits/sec   98    452 KBytes       
[  5]   5.00-6.00   sec  1.06 GBytes  9.14 Gbits/sec    7    447 KBytes       
[  5]   6.00-7.00   sec  1.07 GBytes  9.15 Gbits/sec   72    472 KBytes       
[  5]   7.00-8.00   sec  1.07 GBytes  9.17 Gbits/sec    1    461 KBytes       
[  5]   8.00-9.00   sec  1.07 GBytes  9.23 Gbits/sec   94    535 KBytes       
[  5]   9.00-10.00  sec  1.06 GBytes  9.13 Gbits/sec  179   5.66 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.7 GBytes  9.15 Gbits/sec  946             sender
[  5]   0.00-10.00  sec  10.6 GBytes  9.14 Gbits/sec                  receiver

and here the diganostics of the 2 servers

 

rambo-diagnostics-20250415-1214.zip tardis-diagnostics-20250415-1214.zip

Edited April 15, 20251 yr by comet424

25 minutes ago, comet424 said:

my spinning hds only getting 33megs/s

This isn't unusual for writes to the parity array. Network speed isn't going to magically make the disks faster.

 

https://docs.unraid.net/unraid-os/manual/storage-management/#array-write-modes

auto and probably cuz i dont copy everything it a cache drive..  and just straight to the array.. and for me i have it set for auto for turbo mode

probably why also if i copy a couple files  i cant even watch a movie off the same computer even with 10gig cards  it stutters and stops and here i figured i should be able to do both without stuttering  or not playing...  

if you wanted a media server for streaming to say 5 people  without stuttering  would you want only 1 computer to to have like plex  and it does nothing else  and i guess no parity drives?  

as i trying to get best performance  but i guess you loose best performance to the feature i wanted  different sized hard drives and add hard drive when i can afford it  where as truenas you gotta buy all the drives at first  but i guess it has better performance

 

 

and do you know from the ipfers  what speed am i getting is it 10gig or is 1gig  or is it inbetween  not sure how to understand all that info

Copying files to the array is really taxing on the system, you're going to get hit with IOWait which is why you'd want to use a cache pool. You should do everything you can to not transfer data to the array directly, especially when you're also trying to do media playback/streaming from the array.

Edited April 15, 20251 yr by MowMdown

Reading from the array is faster than writing, since parity isn't updated. And if 2 different streams are reading different HDD that will work better than if both are competing for the same disk.

ok i just have 1  1TB sata SSD for cache drive is that good enough or do you need to have like 2  to mirror them..   and will that speed up copying too if your copying from a pool drive like a download or a cctv drive pool to the array...  as i try to speed up things like with the 10gig nic  but i find if i move files  2 moving  large files and trying to stream 1 video on the same desktop  i didnt think there be any slow downs  as i have 10gig nics  so it would max out the hard drive speeds..  so when copying files to the array is taxing the system  on IO  is that the same for like TrueNas with there raid stuff does it get taxed the same  or is it different?  not like i cant run truenas anyways as i cant afford 3 or 4 large hard drives in one shot  so  that i cant really test  just curious..

 

so if 2 streams are accessing 1 hard drive do you see chunking?  and is there anyway to fix that... do you guys recommend  like moving Media Files to a 2nd server just for watching movies tv shows.. and the main server  keeps a back up of the videos does the backups of kids files,  documents photos etc..

 

and i tried googling.  above about the 10gig stuff i think its saying my 10gig nick only operating at 1gb   as the 9Gbit/s  is should be least 90Gbits  to get close to the 10Gig network  

 

so how do you 2 guys  setup your media  server to get the best out of the system...  like all in one i have or like 2 seperate...  as i have 5 unraid servers i bought 3 are home servers and 2 are off site at sisters house.  this way i have off site backup too..  but always looking to get best speed as i can..  without spending a fortune  on needing the same hard drives like with truenas

Sata SSDs are basically capped to 500MB/s so about 1/3rd of your 10Gbps NIC. You really would need NVME storage if you really want to hit the 10Gbps number.

 

Mirroring Sata SSD in Raid0 can speed it up but I don't think you'll get 1250MB/s(10Gbps) Also depending on what CPU you have, that could also determine how fast your connection is. Hitting those high speeds puts a lot of load on CPUs.

 

Writing data to a POOL outside of the array should prevent your issues. That's how 99% of us use unraid. See my own server setup below:

 

My ingress is straight to my Cache pool which is dual mirrored. Cache and Cache2 are mirrored and Cache3 and Cache4 are mirrored and then the two mirrors are RAID0 together for speed and resiliency. After this my data is sent to the array each week at night when server activity is low. No bottlenecks even if Sabnzbd is downloading and unpacking files.

 

Edited April 15, 20251 yr by MowMdown

my main server  is a ryzen 5800x 8 core cpu and 32gig ecc ram

so my drives are like this in the pic..  but i still find is slow as molasses  even if you play a video off the array  it can be slow even if the drives are spinning and not sleeping

 

so i have 1 drive for Cache sata,  1 drive for Lan Cache, as my internet slow less then 5megbit so i use a lanche..  a download drive for misc downloads  and then i move to array

games drive i was playing with for network share  so you only have to install game 1 location but have a dual boot windows 10 11 and point to it   was playing with it

 

and then i have dual 2tb nvms  for my unraid files..  iso, vms, appdata  as i found plex was always slow and still slow at times...

 

i using a Asus ROG STRIX X570-E motherboard so only has 2 nvmes  on there

 

your setup i take it ur using 4 nvmes?   and if so how do you do it...  

 

and do you know if the iperf test is getting 10 gig cuz i confused about 1.25 and what i get is 1 or its the 9...  

 

and ya its so annoying  when you copy a file  it will even stop the video  or  it wont even allow a video to start in vlc  it will just keep running that yellow line for loading and it may stay stuck 5 min...   till the copying is stuck.. is that all due to the copying straight to the array..

 

i guess truenas doesnt have this issue?  but being free wasnt fezable since i needed to buy 1 drive at a time and ended up buying 5 unraid licenses just so i can go  1 2 drives when they affordable  and now western digital has 32TB drives  could use a couple of those  and just use my 14tb drives for something else lol

 

and would mirror  2 sata ssds improve things? or 4    or i seen  you can put 2 nvme in 1 single sata slot with the enclosure  would that improve things or still capped at 500mbs

There's a lot to unpack here, this post is kinda all over the place so bear with me as I try to organize my thoughts based on the information you've provided. We got different things going on here now.

 

Let's start with the initial concern... To me based on the iperf3 test it sent a total of 10GBytes of data at a speed of 9.15Gbps indicated by the results you've provided. You got your 10Gb/s. We know it works.

 

 

To transfer data to unraid at that speed to a disk drive, you will need to be writing that data to an NVME drive exclusively. Any single disk connected to a SATA port will not be able to hit those speeds. HDDs cap out at around 250MB/s and Sata SSDs cap out at around 500MB/s both well below 1250MB/s (10Gbps) Also keep in mind that to hit 10Gbps it will need to be a single large file transfer with sequential writes. Transferring a bunch of small files will kill the speeds. Just how it works.

 

----

 

For the second portion, you're talking about plex being slow, not really sure I can help you with just that tiny bit of information.  What does "being slow" mean? Media playback off an HDD doesn't need a whole lot of bandwidth even for 4K remuxes (full fat 4K non-encoded media). You can easily stream a few 4K remuxes off an HDD and be nowhere near maxing out the bandwidth of the drive.

 

Hopefully your plex metadata is in your unraid_files pool as that's where thumbnails and stuff area loaded from.

 

11 hours ago, comet424 said:

and ya its so annoying  when you copy a file  it will even stop the video  or  it wont even allow a video to start in vlc  it will just keep running that yellow line for loading and it may stay stuck 5 min...   till the copying is stuck.. is that all due to the copying straight to the array..

 

You shouldn't be copying files to the array directly. If you're doing this, you're going to be fighting parity calculating (reading and writing to two disks) while you're also doing reads, this will cause hiccups for media playback. All your incoming newly written data should be hitting the cache pools first. Mover should handle moving data from cache pools to the array, and you should have this scheduled for a time when server activity is low.

 

Also Im noticing you have HDDs as some of your pools, while there isn't anything wrong with that, you need to keep in mind the speed limits of HDDs. 250MB/s max speeds but only when doing large sequential writes it will be much slower if you're doing something like downloading and unpacking archives.

 

NVMe drives aren't capped at 500MB/s they typically are around 3000MB/s but can be much higher depending on the generation of PCIe. PCIe Gen 5 NVMe speeds can hit nearly 7500MB/s reading and 5000MB/s writing.

 

I think your "issue" is maybe part misunderstanding how unraid works and a misalignment of expectations. I can't really see a physical issue here other than I think you just need to optimize where you're writing data to first.

 

----

 

Here's how I would arrange things. Use the two NVMe drives (unraid_files) for both appdata and file-transfers/downloads from LAN devices. 

 

Use the  "cache_drive" pool drive for downloads from the internet if you're downloading from usenet or torrents. Make this your new download_drive.

 

Keep the HDD pools and your array mostly as they are except for the "download_drive" don't use that one for file downloads because it's going to severely limit your speed. Either find a different purpose for it or maybe consider adding it to the unraid array.

 

Lastly, if you want to add more NVMe drives, you will need to get a PCIe add-in-card that houses NVME drives and you also need to make sure bifurcation is supported in the BIOS and set it to "x4 x4 x4 x4" mode. This will be required to occupy the top PCIe x16 slot (closest to the CPU). This will allow you to add in up to 4 NVME drives and they each get x4 slot allocation.

Edited April 16, 20251 yr by MowMdown