I am not even sure where to post or ask at this point.

I tried posting this question 4 times on Reddit https://www.reddit.com/r/unRAID/

All 4 of my posts were deleted. I don't know who runs that area but I won't go back there. It is toxic.

I have the current version of unRAID up and running finally but have been rather disappointed at just how slow the transfer speed is even with a rather fast Samsung 2tb 9100 Pro NVMe drive used for cache assigned to my shares.

Can someone point in the direction of best tips and tricks to maximize my transfer speed to the array? I am only getting about 200MB/sec currently. I would have thought transfers speeds would have been MUCH higher.

I am copying over 150TB of data from eleven Windows Server 2019 volumes via 10Gbit Ethernet. At this rate it will take weeks to copy this data.

thanks

If you are copying 150TB there's not much point in using the SSD first, since you will then need to wait for the mover to empty it to the array before you can continue.

If you need more speed than the array can sustain, which is always limited by single disk speed, the best option is to create a raidz pool instead. Of course, there are some disadvantages.

Thank you for your reply.

Yes, I've discovered there's more in lost time babysitting the Mover and potential for the 2TB cache drive filling up than just disabling the cache from whatever share I am trying to populate. I'm just going to let the slow transfer take it's course. One of my Win Server Music drive volumes is 26TB. No way am I babysitting unRAID mover during this data copy.

My main reasons for trying unRAID was to have massive volumes that I could expand AND the ability to use different sized disks. I have not checked to see if raidz can accommodate different sized disks. I have nothing else running on this new machine, no VM's, Docker or anything else. I'm not sure I will ever use this for anything other than file server duties. We shall see. I have plenty of hardware lying around. I own and run a computer service company for 20 years (40+ years in I.T.).

My current two Windows 2019 servers (primary and backup) have many different sized disks which I will combine into a 2nd BACKUP unRAID server. I just don't trust 20 years of work on a single machine or array. Right now this unRAID server (3rd server) was built with all new components including ten new 26TB WD Gold drives.

Something that still urks me is just how slow copying data to a 2TB Samsung 9100 Pro SSD is on unRAID. These drives can write over 14GB a second!! Right now they run at SATA HDD speeds or slower. There has GOT to be a reason for this. I just bought a 2nd one which arrived this morning because I want an SSD mirror for redundancy.

Right now the new unRAID server is on my bench for testing. I'm not sure about the bench network cable specs. I have a new pair of Cat 8 shielded cables coming today capable of 40gbit. We shall see if any of this makes a difference.

unRAID machine specs:

24 bay 4U SAS 12gbit backplane hot swap chassis

1000watt eVGA SuperNOVA P5 platinum power supply

Asus Z890 ProArt motherboard

Intel Ultra 5 245K CPU

G.Skill Trident Royal DDR5 6400 RAM 96GB

LSI 9305 16 port SAS controller

Ten WD Gold Enterprise 26TB drives

Two Samsung 2TB 9100 Pro cache pool drives.

60 watt fan controller for massive fan wall server fans, manual speed adjustment.

Intel 710-DA2 Dual port 10GB Ethernet (SFP+ transceivers)

Cyberpower 1500VA PFCLCD UPS backup

unRAID 7.23 Pro lifetime license

16 minutes ago, MHzTweaker said:

I have not checked to see if raidz can accommodate different sized disks.

It can't for now; it should in the near future. You typically can't have it all, I'm afraid every option has advantages and disadvantages, array is much more flexible, and a RAIDZ pool is much faster. You will need to choose based on your priorities. I've used both depending on the use case/server.

18 minutes ago, MHzTweaker said:

Something that still urks me is just how slow copying data to a 2TB Samsung 9100 Pro SSD is on unRAID. These drives can write over 14GB a second!! Right now they run at SATA HDD speeds or slower. There has GOT to be a reason for this.

We can try to debug this. To start, I recommend running a single stream iperf test in one direction, then the other one (-R) and/or post a screenshot from the Windows Explorer graph when transferring a large file. The graph itself can give some clues on the issue.

20 minutes ago, MHzTweaker said:

These drives can write over 14GB a second!!

P.S. not sustained; they are much slower once the pseudo SLC cache fills up, less than 4GB/s IIRC.

Are you using "turbo" (reconstruct write)?

https://docs.unraid.net/unraid-os/using-unraid-to/manage-storage/array/overview/#array-write-modes

Also, if you do the initial data load without parity assigned it will be limited only by the speed of the single disk. Then you can assign and build parity after.

Yes, I am using reconstruct write method and getting 120+ MB/sec. Before changing that setting I was getting like 65MB/sec

I found this trick doing a google search on a Reddit page I think. So many of these things are not obvious to me. I have no idea why a person WOULDN'T want reconstruct write turned on. Are there negatives?

My current situation copying some Sci-Fi movies to the Movie share without using cache.

@JorgeB I will try to figure out how to run this iperf test you requested as soon as this round of copying is done, I also need to install the new Ethernet cables and the 2nd Samsung 9100 cache drive. I also need to flash the motherboard BIOS and go through that with a fine tooth comb making sure settings are proper for unRAID as well. It will be late tonight or tomorrow morning before I can make all these mods.

6 minutes ago, MHzTweaker said:

I have no idea why a person WOULDN'T want reconstruct write turned on. Are there negatives?

The tradeoffs are explained at the link. Main negative is all drives have to spin.

Since I already have most of my data on my drives (WORM - write once, read many), and I don't write more at one time than cache can hold, I can cache many of my user share writes and have them transferred to the array during the night. And often a single drive will be the destination for writes to the array, so no need to have them all spin.

@trurl that's really where I will be once I get all this stuff moved and gain mental confidence I won't lose my data hoard.

Early in the morning and late at night I put new content on my media server which gets scanned by desktop computers running KODI connected to TV's.

Once the stuff is on the server it doesn't really move around. Well, it did once a year on Windows Server 2019 when I found myself outgrowing single disk volumes and having to copy stuff to bigger drives. What a pain that was. It was an event like going to the Dentist for a root canal. That's why I'm here and on unRAID.

With Windows Server 2019, the first 5 gigabytes copied at full 10gbit 900MB/sec to 1000MB/sec network speed, I supposed because of whatever buffer Server 2019 had set aside for writes. unRAID is just a different animal. Maybe I read too much into or assumed the SSD disk cache was like a revolving door and that stuff came in super fast then automatically trickled out to the array as the array could accept it. I had no idea about the "MOVER" task or any of that stuff.

I've installed both new shielded Ethernet cables. Throughput hasn't changed. I will wait for this current round of copying to finish, add the 2nd cache drive as a mirror then try to test some.

I did the parity sync before I started copying data and that took about 35 hours for two 26TB parity disks. I may leave parity off when I do my second one of these in a month.

13 minutes ago, MHzTweaker said:

assumed the SSD disk cache was like a revolving door and that stuff came in super fast then automatically trickled out to the array

It is impossible to move from fast cache to slow array as fast as you can write to fast cache, and trying to move at the same time just makes it worse. Mover is intended for idle time. If you need to write more at one time than cache can hold, don't cache.

That's not great, but also not bad, and not the current issue. Post the Windows Explorer graph when transferring a large file to the SSD.

2 hours ago, JorgeB said:

That's not great, but also not bad, and not the current issue. Post the Windows Explorer graph when transferring a large file to the SSD.

@MHzTweaker, The AI summary from Google search:

Remember that AI is not always 100% right but I know the first one is a factor that most people are completely unaware of.

Edited February 22Feb 22 by Frank1940

I'm leaving the Samsung 9100 Pro cache pool OFF until I finish copying the remaining 100+ TB of data. It makes no sense to use a feature not suited to task.

I'm also not removing the PARITY drives. I've already copied 36TB of the data and do not want to lose more time including the 2 days I spent syncing the pair of 26TB parity drives. I will just tough this out. I have a second unRAID server to build as a mirror image backup of this new unRAID server. I will wait until data has copied before syncing parity when I begin work on the 2nd unRAID build. Since the backup server isn't constantly accessed, the hardware will not be as new but still up to task I hope. I will repurpose either an i7-11700k (socket 1200) or a Xeon E52630L V4 (socket 2011) with 64GB DDR4. Current unRAID server's Intel Ultra 5 245k CPU usage NEVER peaks above 6%. Ever.

Certainly no one in the room believes a 2TB Samsung 9100 Pro PCI express 5.0 drive is limited to 250MB/sec ? Whether the Samsung's DRAM cache is full or not, these drives are capable of more than this. At least 1800MB/sec according to Tom's Hardware review. It is almost like these drives are not being accessed as NVMe devices but SATA/Chipset type devices. Even if so, SATA is capable of 550MB/sec. I have a 2nd 9100 Pro to add when I am done copying data and I'll reconfigure the Cache pool as a mirror for redundancy.

https://www.harddrivebenchmark.net/hdd.php?hdd=Samsung+9100+PRO+1TB+SSD

https://www.tomshardware.com/pc-components/ssds/samsung-9100-pro-ssd-review/2

The SSD Review

Samsung 9100 Pro Gen5 4TB SSD Review - Fastest & Highest...

Isn't competition a great thing?  I have always considered the development of flash technology as somewhat a sport.  It is simply one company trying to creat

I am wondering where unRAID deals with SSD TRIM? Is this automatic? Can I invoke garbage collection manually?

14 minutes ago, MHzTweaker said:

I am wondering where unRAID deals with SSD TRIM? Is this automatic? Can I invoke garbage collection manually?

Settings - Scheduler

@trurl Thanks

37 minutes ago, MHzTweaker said:

@trurl Thanks

Just a hint on using the forum. You have to type the @ followed by enough of the username to get the popup list, then you have to actually select from the list.

Like this

@MHzTweaker

14 hours ago, MHzTweaker said:

Certainly no one in the room believes a 2TB Samsung 9100 Pro PCI express 5.0 drive is limited to 250MB/sec ?

No, I just mentioned that it won't be going full speed all the time, but it should always be much faster than 250MB/s. Please post the Windows Explorer graph when you can.

I may be at least another week. I have four 26TB drives worth of stuff to copy over first before I add the 2nd Samsung and try the cache pool again. It is looking like 2 days per drive. Two are music drives so there's like a million+ files, so miserable small file reads and writes. 😔

Also, I'm hoping this data dump can help validate that things are solid and reliable.

Edited February 23Feb 23 by MHzTweaker

Okay, some progress concerning what is going on and my experiences thus far.

I've copied 138 TB of data so I had a bit less than I had estimated and under the 150 TB I thought existed. Nice

Yesterday I redirected a KODI desktop PC TV node to point to this new unRAID server and it scanned for video content for roughly 7 hours before finishing. That much seems to work correctly. I have 2 more KODI/TV setups to convert to the new server but my point is, unRAID is LIVE and working correctly to meet my needs.

Here's the weird part. My daily grind has me adding content in the morning and evening to the unRAID server. When I was doing the initial content transfer, I mostly got 130MB/sec while transferring this huge hoard. The weird part is while copying individual and a few shows or movies at a time, I am getting full 10Gbit saturation or 1000+ MB/sec while doing these relatively small transfers to the HDD array. This is better than I expected. It would seem as if something is buffering gigabytes of data at a time as I copy. With Windows 2019 Server, I could drag about 6GbB and then the server would have to catch up pausing the copy until data was written. With unRAID I have not figured out what that limit is yet but I like what I am seeing. This helps my work flow. I was afraid I was stuck forever at 130MB/sec while doing my daily administration. NOW, I am very pleased. Keep in mind, I am not using the NVMe SSD cache at all right now. In fact the server and the array has been up for just about 20 days.

Anyone have insights into why after a few weeks the server copy speed is so speedy? It wasn't in the beginning.

This week I will finish making this new server my new main server then decommission the old MAIN and BACKUP Windows servers, stripping out all the largest drives to create a backup unRAID server. Then the 138TB data copy process starts all over again, only this time I will wait to add the 2 parity drives after the copy has finished. It is then I will add the 2nd Samsung 9100 Pro SSD cache drive as a mirror. I must say these 24 bay hot swap chassis's are the Bee's Knees!!! This is end game stuff for me.

1 hour ago, MHzTweaker said:

something is buffering gigabytes of data

Linux does this with any unused RAM.

Are you using an SSD cache/pool drive for your media data. Also Unraid will use (unallocated) RAM as a write buffer which will allow you to approach 10Gbit speeds as you have seen.

I also use KODI and I use a single sql database for all of my KODI clients. I use the mysql Docker for this Database and all of the KODI clients are linked to use it. You can find the basic instructions here:

https://kodi.wiki/view/MySQL/Setting_up_MySQL

The big advantage to doing this is that you only have to build (and maintain) the database once and all of the clients always have the up-to-date information.

It has been several years since I first went down this route. It took several tries to get everything working right. (Cockpit errors on my part mostly!!!) I would recommend using either SQL or MariaDB Dockers for this approach– your choice. It makes handling the database maintenance chore a piece of cake compared to having a separate database on each client!

I'm completely noobish to Linux so it's not a shock to me I do not know what behavior to expect. Nice. I'm stoked I splurged for 96GB DDR5 just before the RAMpocalypse. It seems like I also got through the door with the HDDs before they went unobtanium. I am slowly discovering a wealth of stuff I can do out of the box that Windows server either couldn't or not without a stack of cash $$$ or even more gnashing of teeth.

Wow, centralized KODI !!! Now that would be a huge time saver. Running the database on the same server the files live probably is a massive time save. Having 3 different KODI clients doing new program searches every couple hours seems like a bottleneck. I will look into this right now. Thanks

No, I'm still not using the cache for anything as of yet. For what I am doing, I don't need super speed once the data is on the array. I'm flipping in over in my head what do about hosting my Quickbooks Enterprise company file. Much reading has lead me to the conclusion I either need to run a VM of Windows off the cache or host on a local workstation (do not want to do this).

EDIT: Media is the only contents of the array currently.

Edited March 8Mar 8 by MHzTweaker

6 minutes ago, MHzTweaker said:

Wow, centralized KODI !!! Now that would be a huge time saver. Running the database on the same server the files live probably is a massive time save. Having 3 different KODI clients doing new program searches every couple hours seems like a bottleneck. I will look into this right now. Thanks

Make sure that you set up your Docker image to be only on a SSD cache/pool and that the appdata share is also only on that cache/pool also for best performance.

2 hours ago, MHzTweaker said:

still not using the cache for anything as of yet. For what I am doing, I don't need super speed once the data is on the array.

Ideally, Docker/VM related shares - appdata, domains, system - would have all files on cache or other pool, with nothing on the array, so Docker/VM will perform better, and so array disks can spin down since these files are always open.

If you want more detailed advice about that...

Attach Diagnostics ZIP to your NEXT post in this thread.

Again, I am very frustrated. It seems like the simplest things escape me.

I have created a backup (2nd unRAID server) except THIS time I did not add my parity drives until after I had copied the 140TB of data to it. So that is now complete.

I am trying to add the 2 parity drives but since they are recycled from their previous life in a Windows server, they are GPT partitioned.

I want to repartition them and format them as XFS but I cannot figure out how to do it. No where can I find where this is done. Online docs are of no help either. Why is something so simple, so bloody difficult????

The drives are available, I can add them as parity but they show up as GPT, I do not want this. No manner of taking the array offline gives me any new options. Stuff like this makes me question my sanity.

EDIT: I am to the point now where I just want to pop the drive trays out, put the drives in a SATA dock on my Windows machine and use DiskPart to run a clean command to wipe the partitions. I don't know how to accomplish this in unRAID.

Edited March 28Mar 28 by MHzTweaker