Hi

 

I am using the latest Unraid Version (06.11.05).

 

Since a couple days I have massive performance issues. The copying of data from the main server (TrueNAS) via RSYNC suddenly is slow as hell, in the area of kilobytes. For minutes nothing happens, before it bursts shortly. I tried to switch to Syncthings, but then noticed that even the docker run command for it takes 1h (!!!) to complete. The system is slow as hell. During the docker run also the entire server is not responding to nothing anymore. A complete catastrophy. The software is basically unusable. No changes to TrueNAS or anything else in the network.

 

Need help or I have to curb this thing. Its supposed to be a smooth working backup and not a maintainance hell.

 

Best Regards

please post diagnostics

Freshly created. Edit: Added snapshot of startup performance of Syncthings (see timestamps, its crazy)

mainbackup-diagnostics-20221226-1720.zip

Edited December 26, 20223 yr by Sesshoumaru
Attached Snapshot of Syncthings Performance

Is there a copy going on at the moment? No disks are writing, do you have normal SMB performance?

Yes, the RSync procedure was going on at that moment. 

The SMB performance seemed normal to me. I could access all files on the main server (TrueNas) with good performance (about 800 MBit/s). But RSync reported copy performance of about 3 Mbit/s ... and even worst. between file copy the system takes a break of several minutes. In this time TrueNas idles and Unraid has two cores on 100%.

 

After a reboot, Unraid's docker performace went up from 10min per compose to 10sec again. So something on Unraid must degrade (system was on for 24h before). 

Try using rsync in the command line, and make sure -z flag (compression) is not used.

Using Rsync from command line didn't change a thing.

I switched to Syncthing and the same issue happens again.

I connected the two server directly via a 10Gbe connection, no change, issue happens again.

I configured the main server (TrueNas) to send data to a SMB share on a PC -- streams smoothly wth 375 MB/s.

 

The error is on Unraid side and I am quite sure that the error is on system level. Even running the Parity Checker is down to 250 kb/s read/write performance, which is laughable for an internal operation. I doubt its a problem with the SMB - even if I have no hard reasoning besides als Parity check being hellish slow.

 

Anyhow, the system is basically unusable. It was fine with 6.9.2 - things went haywire with 6.9.15.

Seems like I have to curb the system and switch so something else, like plain server with MergeFS and SnapRaid. A pity...... but the current state is just inacceptble.

I cannot reproduce that or ever seen a similar complain, suggest booting in safe mode and leaving VM and docker services disabled to see if it helps.

I will try that.

What I can defintly say is that docker has a serious issue, because it stopps respoding and as shown previously, runs extremly slow after some writes to the disk happen.

I am puzzled by this problem as well. Its as if once the backup operation starts (regardless of rsync or syncthing) somethng in the system locks up and causes the system to come almost to an entire halt. Docker being unresponsive, SMB being unresponsive, disk read/write going doen in the kb/s area (for disks usually make 100Mb/s without issue).

I am at my wits end.

Switching back to rsync, as docker is off .. same issue. At least two cores are at 100% and the system halts after about 5min, no data transfer .. or stuck, very slow.

I will setup a machine with MergerFS and Snapraid as test, if it works then, server will get culled and switch to that solution.

I've now tested for two days with an Unbuntu based MergerFS + Snapraid solution and the Sync (using Syncthing) is stable with the main TrueNAS server around 180 Mb/s - which basically is the max speed of the backup drive. This will now be my main backup solution for the time being.

 

After a bit lurking around in the logs, I am relatively convinced that the issue lies within the SMB implementation. There is some heavy busy waiting going on in the SMB servicee which occupied 2-3 threads, which do not get a lock on the target device. Only different thing about this drive (except from being Hitachi and not Toshiba like the others) is that it is already filled 99% (80GB free). I gave up further analysis as I have a working backup again.