Strange Write Behaviour

[cptechnology]

Hi All,

First time poster here, just started using UNRAID over the last few days because people seem to rave about it and how easy it is to use, and they are correct in that it is easy to use. The problem I am having is that I get some odd write speed behaviour.

 

Firstly my setup.

I have a HP DL380E Gen 8 with 32GB of Registered ECC RAM

LSI SAS9210-8i HBA card (I need to check if its running in IR or IT mode)

6x 6TB Seagate Barracuda 5400RPM Drives in the array (1 as parity)

1 more 6TB outside the array (I plan to add it to the array once I copy stuff off it)

2x 120GB SSD's as Cache drives.

 

Doing some reading on the forum it seems people are having issues with v6.7.x, so I downgraded to 6.6.7 but my issues still remain.

 

The Problem

Doing an RSYNC from the 6TB outside the array to one of the array disks starts off at about 100MB/s+ but after 10-20 seconds it drops to 20MB/s and then moves around up to 30MB/s all the way down to 500kb/s. These are large files its copying, from 500MB up to 100GB each. The average speeds per file reported by RSYNC range from 7MB/s to 42MB/s.

 

I also copy stuff via the gigabit ethernet from my pc which seems to use the cache drives (rsync does not, I guess because its not going through SMB) and it starts off saturating the network at 112MB/s for a good 30 seconds before it drops down to around 30MB/s and then its up and down like a yoyo, sometimes dropping to 0b/s for about 5 seconds before going back up again.

 

TOP reports iowait of 40%+, iotop shows write speeds jumping all around spiking as high as 250MB/s

 

I understand write speeds will be fairly low with a parity drive, but I don't understand the variability in the write speeds, especially the start-and-stop nature of them.

My previous NAS was running on an old i7-930 with 16gb RAM, using OpenMediaServer and it would give me read and write speeds that saturate my gigabit network consistently. Of course that is a completely different system and not on the DL380.

 

I have attached the diagnostics, and I would appreciate any help.

 

pegasus-diagnostics-20190905-2209.zip

[trurl]

1 hour ago, cptechnology said:

I also copy stuff via the gigabit ethernet from my pc which seems to use the cache drives (rsync does not, I guess because its not going through SMB)

No that's not right. If you are writing to a cached user share it will go to cache regardless, unless you have too little space remaining on cache (cache Minimum Free is in Global Share Settings). Some sort of confusion with this part. Maybe you aren't writing to a cached user share for one of the cases you tested. If you tell it to write to an array disk that is what it will do. Only writes to cached user shares are cached.

 

 

[cptechnology]

The share it was writing to was set to use Cache, and the cache storage/write info never changed while rsync was running. I had another issue last night where it was copying files via the network but after the cache drive filled up windows reported that the drive was full and wouldn't let the process continue to transfer.

[trurl]

23 minutes ago, cptechnology said:

after the cache drive filled up windows reported that the drive was full and wouldn't let the process continue to transfer.

Unraid has no way of knowing how large a file will become when it decides what disk to write it to. You must set Minimum Free to larger than the largest file you expect to write. If a disk has less than Minimum Free, then Unraid will choose a different disk when it begins to write the file.

 

If you try to write a file and Minimum Free agrees that the disk can be chosen, then Unraid may choose the disk, and if it doesn't have enough space for the file, the write will fail.

 

Cache has Minimum Free (in Global Share Settings), which Unraid will use to decide whether to write to the array instead of cache when you are writing to a cache-yes or cache-prefer User Share. Cache-only shares will only go to cache, and cache-no shares will not go to cache.

 

Each User Share also has a setting for Minimum Free, which Unraid will use to decide which disk to use when writing to the share. This comes into play even if the share isn't cached. User Shares can span disks in the array, and if an array disk has less than Minimum Free for the user share, Unraid will choose a different disk.

 

There is Help in the webUI which can be toggled on/off with Help (?) on the top bar, and you can also toggle help for a specific setting by clicking on its label.

 

There are other nuances to the various Use cache settings explained in more detail here:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#comment-537383

 

[cptechnology]

I know I haven't been using UNRAID for very long, and I am sure there are reasons for it being programmed the way it has, but having to make a guess at what the largest file might be so UNRAID doesn't try and write to a full disk seems rather primitive. I have not come across any other software that requires that, and this is paid software.

[trurl]

If you open a file for writing, how is it supposed to know how large the file will become? What other systems are you aware of that will let you write more than a disk will hold?

[cptechnology]

I guess what I mean is that most other systems seem to use more of a RAID 50 config, striping over the raid array so there is no need to know how large the file will be for that given drive. I got fed up with unraid and its extremely poor performance last night and ended up trying FreeNAS. It was more involved in setting up but now I can get 80-90MB/s write speeds consistently. It appears to stripe the data over the array.

[JonathanM]

5 minutes ago, cptechnology said:

I guess what I mean is that most other systems seem to use more of a RAID 50 config, striping over the raid array so there is no need to know how large the file will be for that given drive. I got fed up with unraid and its extremely poor performance last night and ended up trying FreeNAS. It was more involved in setting up but now I can get 80-90MB/s write speeds consistently. It appears to stripe the data over the array.

Striping gives speed and single volume, at the expense of flexible expansion with different drive sizes and the ability to read each disk independently. If a RAID 50 exceeds failure redundancy, all data is lost. If unraid exceeds failure redundancy, only the failed drives are lost, the rest of the drives are fine. There are pros and cons to each, but to me and everyone else actively using unraid, the benefits far outweigh the negatives.

 

As far as extremely poor performance, that's likely due to hardware or some configuration issue. 1Gb speeds are attainable with the right setup.

[cptechnology]

Unfortunately despite spending 2 days going through forum posts and trying different configs, I was never able to achieve anymore than 30MB/s on unraid

[remotevisitor]

Did you try enabling ‘reconstruct write’ mode (aka ‘turbo write’) to see if that improves your performance on your system?