Am I just expecting the wrong capability from unRAID?

[rockrimmon]

I just finished building an unRAID server and I am wondering if my intention of how to use the server is fundamentally flawed?

 

My intention was to make the server the target for four Mac computers for Time Machine backups and/or third party backups (I have tried Carbon Copy Cloner, SuperDuper!, Chronosync, Yoyotta and simple Finder drag and drop so far in addition to Time Machine). I also copied from an external USB drive 12TB of all of my photography (connected the external drive direct to the server and used Krusader to copy from the USB drive to the server). 

 

As seen below the equipment in the server should be fairly robust. All drives are connected to two LSI PCIe adapters, the HDDs connected to one of the LSI boards, the cache on the second LSI PCIe board. The server has been rock solid, it has been up over 8 days since first power on after the build, no crashes, no reboots, zero errors on any disc. Drives are running typically in the mid 30 to low 40 degrees C, hottest ever was a drive that got to 48 degrees C (I am changing the cooling in the case to address that). CPU has never gone over 38 degrees C. unRAID is version 6.9.2.

 

What I have discovered is that trying to copy the entire contents of any of my Macs to the server, whether via Time Machine, or any of the other alternatives mentioned above is abysmally slow, average transfer rates are ranging between 300Kb/s to 240Mb/s with most of the network transfers in the less than 2Mb/s range, these tests are all on a 10GbE network, to the point of not even being a viable option. I was prepared for the first Time Machine backup to take a long time, although 40+ hours was beyond what I expected for less than 1TB of data on the client. The issue with Time Machine is that every subsequent incremental backup is taking more than an hour in general, thus hourly backups cannot happen effectively. For any of the other options I have tried even after 48+ hours none of the other options have even reached 50% completion of a source machine with just 700 GB to be copied. In fact, Chronosync (which normally is quite speedy to a locally attached external drive, even standard old USB 2.0 is usable) I have had running over 24 hours to just copy the Applications folder of one of my Macs to the unRAID server as a test. This Application folder has 80GB in it, and after 24 hours is nowhere near complete (based on the number of Applications already copied to the server, I would say less than 50% of the 80GB Applications folder has been copied in over 24 hours).

 

I have tried two different clients, an iMac Pro and a M1 MacBook Pro. The iMac is using its internal 10GbE connection, and I have a Thunderbolt to SFP+ 10GbE adapter on the M1 MBP, and the server is connected via SFP+ DAC from the 10GbE adapter in the server to the same 10GbE switch the iMac Pro and M1 MBP are connected to. 

 

I have applied all of the tuning suggestions in Spaceinvader One's videos, I have verified flow control is disabled. CPU utilization on the server has never exceeded 51%, and in most cases is in the 20% or less range, memory utilization is right at 11% consistently, I have all shares set to use cache, the cache pool is two Crucial MX500 1TB SSDs.

 

I have run iPerf3 in all combinations of what machine is client versus server and consistently see 9.3Gb/s, or higher, so I know the network appears to be solid. I created a 10GB single file filled with zeros and it transfers at well over 2Gb/s from client to server. But any attempt to do a full system backup, where there are 1000's of small files crawls so badly as to be unusable. 

 

So, the question is, was I just way out of line with my expectations of being able to use the unRAID server as a target for hourly incremental backups (assuming I can even get the initial full backup to ever complete)? I am wondering if I should have gone with TrueNAS, but I really like the UI of unRAID and the apps and dockers that are available. Based on what I am seeing with respect to files hitting the cache first, and cache never reaching capacity, and given the server performance numbers, I suspect that even going to something like TrueNAS will not significantly change anything, other than the multiple spindles to write to given the distributed parity in TrueNAS and ZFS, I am just not sure if that is really where the bottle neck is, or if this is just a fundamental limitation of the reality of trying to backup numerous small files with a NAS as the target using SMB? The diagnostic files have been attached to this post.

 

Appreciate any ideas, or verification if anyone is using an unRAID server as I am attempting to?

 

The server equipment is as follows:

 

Supermicro X9SCL/X9SCM, Version 1.11A

Intel® Xeon® CPU E3-1270 V2 @ 3.50GHz

32 GB DDR3 ECC memory

Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection

Parity 1 ST18000NM000J-2TV103_ZR509DE8 - 18 TB (sdb)

Parity 2 ST18000NM000J-2TV103_ZR503DR8 - 18 TB (sdc)

Disk 1 ST18000NM000J-2TV103_ZR503L20 - 18 TB (sdd)

Disk 2 ST18000NM000J-2TV103_ZR503NPW - 18 TB (sde)

Disk 3 ST18000NM000J-2TV103_ZR502BLR - 18 TB (sdf)

Cache CT1000MX500SSD1_2152E5F7C26A - 1 TB (sdg)

Cache 2CT1000MX500SSD1_2152E5F7C283 - 1 TB (sdl) Part of cache pool

Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

 

Share configuration:

appdata Prefer : Cache

DaveDownloads Yes : Cache

DaveiPhoneVideos Yes : Cache

DaveiPhotoExports Yes : Cache

DaveMacStudioClone Yes : Cache

DaveMacStudioTimeMachine Yes : Cache

DaveMBPClone Yes : Cache

DaveMBPTimeMachine Yes : Cache

DaveMovies Yes : Cache

DavePictures Yes : Cache

DaveWebSites Yes : Cache

domains Prefer : Cache

isos Yes : Cache

system Prefer : Cache

 

lv426-diagnostics-20220504-0836.zip

 

 

Edited May 4, 2022 by trurl
delete lots of attached files that should have been a single zip

[trurl]

Cleaned that up for everyone. I assume you figured out you should only attach the single zip file that was downloaded.

[wgstarks]

Getting macOS to work well with SMB can be a bit of a challenge. There is a fair amount of discussion on this forum in various threads regarding the best SMB settings (settings>SMB>SMB extras). Here is what is working well for me-

#macOS config start
   server multi channel support = no
[global]
   vfs objects = catia fruit streams_xattr
   fruit:nfs_aces = no
   fruit:zero_file_id = yes
   fruit:metadata = stream
   fruit:encoding = native
   spotlight backend = tracker

[Media]
   path = /mnt/user/Media
   veto files = /._*/.DS_Store/
   delete veto files = yes
   spotlight = yes

[mini_SMB]
   path = /mnt/user/mini_SMB
   veto files = /._*/.DS_Store/
   delete veto files = yes
   spotlight = yes
#macOS config end

[global] is applied to all shares.

[Media] and [mini_SMB] are shares on my server that I have applied the associated settings to. Edit the names in the square brackets to match your shares. You can also apply those settings in [global] if you want to use them everywhere. I haven’t tested that though so no idea what unintended consequences might come up.

 

If you are running unRAID 6.10 rc5 or later you don’t need server multi channel support = no. That’s already the default.

 

Edit: You also need to adjust the path in per-share settings to match yours.

Edited May 5, 2022 by wgstarks

[kram96]

Has anyone ever resolved the Mac SMB issue? I confirmed it is Mac SMB only.

The workaround for me is NFS. Connecting with nfs://SERVERIP:/mnt/user/SHARE