[Worrisome performance issues]

Appreciate the help, I think what might work best for this (massive amount of initial data onboarding - trying to decom all my old NAS units and other random storage stuff) would be disabling parity, and then just doing rsyncs specifically to different disks, and then later adding back in parity and letting it churn for the 20TB each parity disk

 

this would allow me to maximize the speed to each individual disk for this initial load.

[Worrisome performance issues]

18 minutes ago, u0126 said:

also in a slightly related question - could files be transferred directly to /mnt/diskX/whatever - obviously unprotected at first - and then run a full parity scan when all that is done and let it do its thing? or do files have to actually pass through the shfs "merged" filesystem layer to be included in the array?

 

after a quick search it sounds like the answer is it's okay if the steps are: "disabling parity, writing to all the disks, then re-enabling parity and letting it build" - because the parity system / disks could possibly mess with /mnt/diskX if they're active, so the solution is removing parity options altogether, then starting from scratch. makes sense.

 

I've got two parity disks - I'm assuming I can stop the array, remove BOTH at the same time, do whatever I want to do to get data onboarded, and then add them back - do I have to do one at a time (so it'll be 5-7 days of rebuild each disk, if I recall my last parity disk addition) - is there a way to add both at once?

[Worrisome performance issues]

also in a slightly related question - could files be transferred directly to /mnt/diskX/whatever - obviously unprotected at first - and then run a full parity scan when all that is done and let it do its thing? or do files have to actually pass through the shfs "merged" filesystem layer to be included in the array?

[Worrisome performance issues]

5 minutes ago, JorgeB said:

You want to use a single rsync session at a time, multiple simultaneous writes to Unraid array will always be slow because of how parity works, and enable turbo write.

 

I'll have to look at turbo write. I'm assuming that doesn't require the cache disk but rather tunes something in shfs (as I mentioned above with cache disk issues) - does it require a reboot, the array to be stopped/restarted, anything? Or just simply change to "reconstruct write" and move on?

 

5 minutes ago, JorgeB said:

Performance is not the main reason we don't recommend USB, it's mostly a reliability problem since they are prone to disconnects and bad at error handling in general.

 

For sure. This isn't going to be a setup with any physical disconnects planned, USB was simply chosen as the connection for a more flexible setup than a large fixed server. As far as error handling I'd expect it to be a detected failure and stop the array/be caught like any other error, just run the risk of it being more often in theory than directly on an internal SATA connection.

[Worrisome performance issues]

8 minutes ago, JorgeB said:

Does this mean you are writing to 3 disks at the same time? You don't want to do that with parity, it will be extra slow, also look a turbo write for initial server load, but note that USB is not recommended for array and pool devices.

 

I'm writing to /mnt/user shares. I'm showing the behavior of what htop and the I/O subsystem is reporting. I'm not trying to subvert Unraid at all.

 

Note that this is a pretty premium USB setup and shouldn't perform much differently than directly being attached to an HBA. I would expect minimal impact (look at the disk performance benchmarks posted) and I've ensured that every component is 3.1 (10Gbps) or better.

 

[Worrisome performance issues]

I've been spending the last week transferring all the data I possibly can over to my new unraid server. At first it all seemed great, except I've been seeing some pretty bad performance (in general - not just samba, NFS, etc... even local it locks up)

 

Unraid version 6.11.5

 

Hardware setup is a bit exotic:

• Intel 11th gen NUC11PAHi7 (11th Gen Intel® Core™ i7-1165G7 @ 2.80GHz)
• 64 gigs RAM
• cache disk - Corsair MP600 GS 2TB PCIe Gen4 x4 NVMe M.2 SSD - High-Density TLC NAND - M.2 2280
• network: 2.5 gig (mtu 1500 still. considering looking into jumbo frames at some point)
• data disks: 10 Western Digital 20TB WD Red Pro NAS HDD - 7200 RPM, SATA 6 Gb/s, CMR, 512 MB Cache, 3.5" - WD201KFGX - 2 of them are for parity.

 

I have a USB 3.2 hub capable of full speed (10+ Gbps) capability that attaches USB C to Oyen Digital Mobius Pro 5C 5-Bay USB-C External Drive Enclosure (3N5-C-M) enclosures - 5 disks in each enclosure.

 

Everything is USB 3.2 gen 1 or better (or USB 3.1 gen 2, it's the same) and can support 10+ Gbps down to the disk (the disks should be the slowest component at 6 Gb/s) - individually hdparm reports each disk almost all exactly capable of this:

 

Timing cached reads: 22554 MB in 2.00 seconds = 11296.57 MB/sec
Timing buffered disk reads: 794 MB in 3.01 seconds = 264.06 MB/sec

 

Now, the share I have setup currently is set to Cache: No, because when I had Cache: Prefer the mover was not invoking (turns out I needed to enable/disable it again after reading someone else having the issue. I set it up the way I wanted to at first, but I filled the cache disk almost immediately even with mover scheduled hourly. Turns out it wasn't aware it was supposed to be doing anything) - but the performance of the mover was pretty bad as well.

 

It seems like I'm hammering these disks from 3 rsync over SSH streams (originally I was trying to rsync over NFS, but it kept locking up, this at least stays consistently moving) - however I'm nowhere near taxing the network, the memory or the disk I/O individually. Everything points to shfs overhead doing something crazy.

 

I've had htop running for a couple days and watching it, and I see crazy ups and downs (see attachment) that's one of the crazy bursts, where it seems like all the processes are doing awesome with the stuff they're supposed to do. But then I'll have periods where only a couple are even over 1 M/sec.

 

Meanwhile, the combined traffic of those 3 rsync streams (the only data that should be changing, I'm not running any active Docker or apps stuff that isn't some basic background/unraid system stuff) is nowhere near that. I'm getting maximum maybe 40M/sec combined.

I'm a bit worried I'm going to take weeks getting all my data into unraid and not have an ability to switch to any other kind of software (I can't imagine it's hardware, I made sure every piece of this was about as good as it could be for a USB DAS based system) and I've got benchmarks on individual components showing they're capable of it. Something is happening in the data streams to/from the disks.

 

Happy to listen to any feedback or share anything I may be missing. Also, it's not just writing to disk, everything locks up. I was trying to ls /mnt/user/some/directory and it stalled for a few minutes before finally returning back. It's like the entire /mnt/user share is totally bottlenecked. iostat also shows multiple devices at near 100% utilization - but not all of them. If there's any tuning or diagnostics or debugging I can do I'd be happy to. I really want this to work. I have 6 of these enclosures (which would magically give me the unraid 30 disk max!) and intend on shucking my externals to add to it and moving everything onto this one massive array.

 

Thanks in advance!