Worrisome performance issues

[u0126]

I've been spending the last week transferring all the data I possibly can over to my new unraid server. At first it all seemed great, except I've been seeing some pretty bad performance (in general - not just samba, NFS, etc... even local it locks up)

 

Unraid version 6.11.5

 

Hardware setup is a bit exotic:

• Intel 11th gen NUC11PAHi7 (11th Gen Intel® Core™ i7-1165G7 @ 2.80GHz)
• 64 gigs RAM
• cache disk - Corsair MP600 GS 2TB PCIe Gen4 x4 NVMe M.2 SSD - High-Density TLC NAND - M.2 2280
• network: 2.5 gig (mtu 1500 still. considering looking into jumbo frames at some point)
• data disks: 10 Western Digital 20TB WD Red Pro NAS HDD - 7200 RPM, SATA 6 Gb/s, CMR, 512 MB Cache, 3.5" - WD201KFGX - 2 of them are for parity.

 

I have a USB 3.2 hub capable of full speed (10+ Gbps) capability that attaches USB C to Oyen Digital Mobius Pro 5C 5-Bay USB-C External Drive Enclosure (3N5-C-M) enclosures - 5 disks in each enclosure.

 

Everything is USB 3.2 gen 1 or better (or USB 3.1 gen 2, it's the same) and can support 10+ Gbps down to the disk (the disks should be the slowest component at 6 Gb/s) - individually hdparm reports each disk almost all exactly capable of this:

 

Timing cached reads: 22554 MB in 2.00 seconds = 11296.57 MB/sec
Timing buffered disk reads: 794 MB in 3.01 seconds = 264.06 MB/sec

 

Now, the share I have setup currently is set to Cache: No, because when I had Cache: Prefer the mover was not invoking (turns out I needed to enable/disable it again after reading someone else having the issue. I set it up the way I wanted to at first, but I filled the cache disk almost immediately even with mover scheduled hourly. Turns out it wasn't aware it was supposed to be doing anything) - but the performance of the mover was pretty bad as well.

 

It seems like I'm hammering these disks from 3 rsync over SSH streams (originally I was trying to rsync over NFS, but it kept locking up, this at least stays consistently moving) - however I'm nowhere near taxing the network, the memory or the disk I/O individually. Everything points to shfs overhead doing something crazy.

 

I've had htop running for a couple days and watching it, and I see crazy ups and downs (see attachment) that's one of the crazy bursts, where it seems like all the processes are doing awesome with the stuff they're supposed to do. But then I'll have periods where only a couple are even over 1 M/sec.

 

Meanwhile, the combined traffic of those 3 rsync streams (the only data that should be changing, I'm not running any active Docker or apps stuff that isn't some basic background/unraid system stuff) is nowhere near that. I'm getting maximum maybe 40M/sec combined.

I'm a bit worried I'm going to take weeks getting all my data into unraid and not have an ability to switch to any other kind of software (I can't imagine it's hardware, I made sure every piece of this was about as good as it could be for a USB DAS based system) and I've got benchmarks on individual components showing they're capable of it. Something is happening in the data streams to/from the disks.

 

Happy to listen to any feedback or share anything I may be missing. Also, it's not just writing to disk, everything locks up. I was trying to ls /mnt/user/some/directory and it stalled for a few minutes before finally returning back. It's like the entire /mnt/user share is totally bottlenecked. iostat also shows multiple devices at near 100% utilization - but not all of them. If there's any tuning or diagnostics or debugging I can do I'd be happy to. I really want this to work. I have 6 of these enclosures (which would magically give me the unraid 30 disk max!) and intend on shucking my externals to add to it and moving everything onto this one massive array.

 

Thanks in advance!

Edited February 14, 2023 by u0126
fixed image

[JorgeB]

6 hours ago, u0126 said:

It seems like I'm hammering these disks from 3 rsync over SSH streams

Does this mean you are writing to 3 disks at the same time? You don't want to do that with parity, it will be extra slow, also look a turbo write for initial server load, but note that USB is not recommended for array and pool devices.

[u0126]

8 minutes ago, JorgeB said:

Does this mean you are writing to 3 disks at the same time? You don't want to do that with parity, it will be extra slow, also look a turbo write for initial server load, but note that USB is not recommended for array and pool devices.

 

I'm writing to /mnt/user shares. I'm showing the behavior of what htop and the I/O subsystem is reporting. I'm not trying to subvert Unraid at all.

 

Note that this is a pretty premium USB setup and shouldn't perform much differently than directly being attached to an HBA. I would expect minimal impact (look at the disk performance benchmarks posted) and I've ensured that every component is 3.1 (10Gbps) or better.

 

[JorgeB]

1 minute ago, u0126 said:

I'm writing to /mnt/user shares.

You want to use a single rsync session at a time, multiple simultaneous writes to Unraid array will always be slow because of how parity works, and enable turbo write.

 

2 minutes ago, u0126 said:

Note that this is a pretty premium USB setup

Performance is not the main reason we don't recommend USB, it's mostly a reliability problem since they are prone to disconnects and bad at error handling in general.

[u0126]

5 minutes ago, JorgeB said:

You want to use a single rsync session at a time, multiple simultaneous writes to Unraid array will always be slow because of how parity works, and enable turbo write.

 

I'll have to look at turbo write. I'm assuming that doesn't require the cache disk but rather tunes something in shfs (as I mentioned above with cache disk issues) - does it require a reboot, the array to be stopped/restarted, anything? Or just simply change to "reconstruct write" and move on?

 

5 minutes ago, JorgeB said:

Performance is not the main reason we don't recommend USB, it's mostly a reliability problem since they are prone to disconnects and bad at error handling in general.

 

For sure. This isn't going to be a setup with any physical disconnects planned, USB was simply chosen as the connection for a more flexible setup than a large fixed server. As far as error handling I'd expect it to be a detected failure and stop the array/be caught like any other error, just run the risk of it being more often in theory than directly on an internal SATA connection.

[u0126]

also in a slightly related question - could files be transferred directly to /mnt/diskX/whatever - obviously unprotected at first - and then run a full parity scan when all that is done and let it do its thing? or do files have to actually pass through the shfs "merged" filesystem layer to be included in the array?

[u0126]

18 minutes ago, u0126 said:

also in a slightly related question - could files be transferred directly to /mnt/diskX/whatever - obviously unprotected at first - and then run a full parity scan when all that is done and let it do its thing? or do files have to actually pass through the shfs "merged" filesystem layer to be included in the array?

 

after a quick search it sounds like the answer is it's okay if the steps are: "disabling parity, writing to all the disks, then re-enabling parity and letting it build" - because the parity system / disks could possibly mess with /mnt/diskX if they're active, so the solution is removing parity options altogether, then starting from scratch. makes sense.

 

I've got two parity disks - I'm assuming I can stop the array, remove BOTH at the same time, do whatever I want to do to get data onboarded, and then add them back - do I have to do one at a time (so it'll be 5-7 days of rebuild each disk, if I recall my last parity disk addition) - is there a way to add both at once?

Edited February 14, 2023 by u0126

[itimpi]

59 minutes ago, u0126 said:

also in a slightly related question - could files be transferred directly to /mnt/diskX/whatever - obviously unprotected at first -

You can transfer directly to a given drive if you have Disk Shares enabled.   They will only be unprotected if you do not have a parity drive as if you do then parity is still updated in real time as you write to the drive.   If you are writing to multiple drives like this and have parity assigned then you still want to avoid multiple streams at the same time to avoid contention on the parity drive.

 

WARNING:  Never mix a disk share and a user share in the same copy/move command as this can easily lead to data loss.

[JorgeB]

1 hour ago, u0126 said:

Or just simply change to "reconstruct write" and move on?

That's it, it doesn't tune shfs, it's just a different way of updating parity, it's much faster (if there are no controller bottlenecks) at the expense off all disks being spun up for any writes.

 

1 hour ago, u0126 said:

could files be transferred directly to /mnt/diskX/whatever - obviously unprotected at first

Yes, parity will still be updated real time.

[itimpi]

47 minutes ago, u0126 said:

I've got two parity disks - I'm assuming I can stop the array, remove BOTH at the same time, do whatever I want to do to get data onboarded, and then add them back - do I have to do one at a time (so it'll be 5-7 days of rebuild each disk, if I recall my last parity disk addition) - is there a way to add both at once?

You can add both at once if you want and the building of parity on them will run in parallel.   In terms of time assume something like 2-3 hours per TB with the total time dictated by the largest parity drive (if not the same size).

[JorgeB]

48 minutes ago, u0126 said:

I've got two parity disks - I'm assuming I can stop the array, remove BOTH at the same time, do whatever I want to do to get data onboarded, and then add them back

You can, just unassign both and start the array.

[u0126]

Appreciate the help, I think what might work best for this (massive amount of initial data onboarding - trying to decom all my old NAS units and other random storage stuff) would be disabling parity, and then just doing rsyncs specifically to different disks, and then later adding back in parity and letting it churn for the 20TB each parity disk

 

this would allow me to maximize the speed to each individual disk for this initial load.

[itimpi]

1 minute ago, u0126 said:

and letting it churn for the 20TB each parity disk

With parity disks of that size you may well want to install the Parity Check Tuning plugin to help alleviate the impact/pain of very long parity checks on day-to-day running.

[u0126]

Just now, itimpi said:

With parity disks of that size you may well want to install the Parity Check Tuning plugin to help alleviate the impact/pain of very long parity checks on day-to-day running.

 

thanks for the tip. for the most part I'm fine with letting it run 24/7 until it's complete, but it sounds like this might help if I absolutely had to pause/tune it.

[u0126]

one thing I've noticed even when not doing much is I get this when trying to stop the array or reboot

 

Feb 14 02:59:58 unraid  emhttpd: shcmd (1457): umount /mnt/user
Feb 14 02:59:58 unraid root: umount: /mnt/user: target is busy.
Feb 14 02:59:58 unraid  emhttpd: shcmd (1457): exit status: 32
Feb 14 02:59:58 unraid  emhttpd: shcmd (1458): rmdir /mnt/user
Feb 14 02:59:58 unraid root: rmdir: failed to remove '/mnt/user': Device or resource busy
Feb 14 02:59:58 unraid  emhttpd: shcmd (1458): exit status: 1
Feb 14 02:59:58 unraid  emhttpd: shcmd (1460): /usr/local/sbin/update_cron
Feb 14 02:59:58 unraid  emhttpd: Retry unmounting user share(s)...

 

just stays in a loop

 

lsof /mnt/user gives me nothing

 

if I manually umount -l /mnt/user then it instantly is able to push past that error. then /mnt/diskX ones also claim to be busy.

 

lsof on one of those shows the "shfs" process still busy on them. looks like it's still doing some sort of parity stuff trying to catch up(?)

 

 

Edited February 14, 2023 by u0126

[u0126]

hmm when I go to try to unassign parity disks in the drop down, it shows as an option, but then refreshes the page immediately. won't let me actually save anything unassigned. I also see in syslog each time I try:

Feb 14 03:07:30 unraid  emhttpd: shcmd (1676): rmmod md-mod
Feb 14 03:07:30 unraid root: rmmod: ERROR: Module md_mod is in use
Feb 14 03:07:30 unraid  emhttpd: shcmd (1676): exit status: 1

 

Edited February 14, 2023 by u0126

[JorgeB]

26 minutes ago, u0126 said:

failed to remove '/mnt/user': Device or resource busy

This means something is still using /mnt/user, an opened SSH session to /mnt/user for example will prevent it from unmounting.

[u0126]

5 minutes ago, JorgeB said:

This means something is still using /mnt/user, an opened SSH session to /mnt/user for example will prevent it from unmounting.

 

yeah I didn't have anything obvious, all rsyncs were killed, didn't see any processes still open, no dockers, no VMs, only SSH was me as root directly into /root. it looked like shfs (as I posted later) still had some /mnt/diskX uses that weren't dying down. like some fuse leftover stuff.