Hi all,

 

Looking for some help on an issue I have now seen across both an N54L and Gen8 microserver.

 

When copying files to the array from a windows PC connected via Gigabit ethernet the writes will start at 100MB/s then drop down to almost 0MB/s then jump to between 5 and 15MB/s. The drop outs then cycle every 20 seconds or so until the file is fully transferred.

 

Trying to decipher the issue, I used ftp via Filezilla from the same PC. Transferring to the /mnt/user/ folder I get the same behaviour. The writes start very fast then drop to almost nothing with time-outs occurring intermittently.

However, transferring direct to the target disk or to /mnt/user0, the transfers start at 100MB/s then continue at around 25MB/s to the end of the transfer with no hangs.

 

As such, I believe the issue is coming from the cache pool. I confirmed this by writing direct to /mnt/cache and saw the hanging behaviour again.

 

The log for unraid (accessed via the GUI) shows no errors.

Both cache drives are connected to a BR10i configured in IT mode via LSI firmware (as described elsewhere on this forum). Both drives are connected to the same miniSAS --> 4 x Sata connector. Could there be a bottleneck here? I would expect to see low speeds and not hangs if this were the case?

I guess the other option is potentially a failing drive?

 

Can someone help me to work out the issue? At the moment I'm tempted to disable the cache as it's making the array almost unusable due to the hanging behaviour.

 

Thanks in advance,

Phil

Can someone help with the commands to check read and write speeds on each drive please? I've seen them used in other threads but can't find it!

 

Thanks, Phil

Both drives are connected to the same miniSAS --> 4 x Sata connector. Could there be a bottleneck here? I would expect to see low speeds and not hangs if this were the case?

 

This won't be a problem.

 

I know unRAID as performance issues with some older controllers, don't now if the BR10i is one of them.

 

Attached is a script to test write speed, I got it from the forum, not sure from who (Weebotech?).

 

Unzip to flash root and type:

/boot/write_speed_test.sh /mnt/diskX/test.dat

 

 

It writes a 10GB test file, you may want to increase that value if you have lots of RAM.

write_speed_test.zip

OK,

 

So I've run hdparm -Tt on each drive with the following results:

/dev/sde: (Cache Drive 1)
Timing cached reads:   11508 MB in  2.00 seconds = 5757.43 MB/sec
Timing buffered disk reads: 796 MB in  3.00 seconds = 265.13 MB/sec
root@Tower:~# hdparm -Tt /dev/sdf

/dev/sdf: (Cache Drive 2)
Timing cached reads:   7742 MB in  2.00 seconds = 3872.25 MB/sec
Timing buffered disk reads: 760 MB in  3.00 seconds = 253.07 MB/sec
root@Tower:~# hdparm -Tt /dev/sdg

/dev/sdh: (240GB SSD On Same Controller - disk3 below)
Timing cached reads:   11544 MB in  2.00 seconds = 5775.54 MB/sec
Timing buffered disk reads: 730 MB in  3.00 seconds = 243.12 MB/sec

So read speeds look OK (over a 3 second period).

 

Just run the write_speed_test.sh to write the 10GB file to:

/mnt/cache - cache pool of 2 x 128GB drives on BR10i

/mnt/disk3  - 240GB SSD on same BR10i as cache drives

/mnt/disk4  - WD RED 3TB on motherboard controller

 

Results were:

Write to /mnt/cache took 13.5 mins (12.6MB/s) with multiple periods where either slowdowns or hanging of data writing occured - e.g. stuck at 9.0GB written for >60s.

Write to /mnt/disk4 took 6 minutes (28.3MB/s) - constant speed

Write to /mnt/disk 3 took 5m 40s (30.1MB/s) - constant speed

 

So definitely a problem somewhere with the cache pool. How do I now start to work out the cause of the problem?

NB: I have attached diagnostic zips to this post.

 

EDIT: I ran another few runs using write_speed_test.sh and copied the output from the terminal into excel and plotted the write speeds (both the speed reported by dd as well as a value I calculated manually from the terminal data) over time. I have attached these as an image and it's pretty telling and confirms a problem in the cache pool.

 

Thanks,

Phil

tower-diagnostics-20160518-2029.zip

I haven't looked at the diagnostics, but there are 2 possibilities that I can think of, one is that trimming is 'messed up' (poor terminology!) on one or more of the cache drives, and the other is that there is file system corruption in the BTRFS of the pool.  I don't know much about trimming, very little experience, so perhaps someone else can help.  And I don't have much experience with BTRFS repair, but I wrote up a section here ->  Drives formatted with BTRFS.  Try the scrub command.  If it doesn't help, you may want to consider redoing the pool, copying all data off, breaking the pool, reformatting them, rebuilding the pool, and restoring the data.

 

But it could just be a trimming problem.  I don't know how you test for that.

 

I assume you have checked SMART reports for each of the drives?  (I really should take the time to check the diagnostics!)

If you don't do regular trims, it's easy to check if it's part ot the problem, run:

 

fstrim -v /mnt/cache

Hi Rob,

 

I've run the scrub command to no effect sadly. I will definitely consider breaking the pool and rebuilding. Maybe worth waiting to see if I can resolve any TRIM issues.

 

SMART reports look OK, one of the drives has a single reallocated sector. I checked this when it was first added to the array and it appears to be common for there to be reallocated sectors with the model I'm using (Sandisk SSD). The rellocated sector count has been stable since it was added to the array (6 months+). The other drive is all green w.r.t. SMART.

 

If you don't do regular trims, it's easy to check if it's part ot the problem, run:

 

fstrim -v /mnt/cache

I get this report back when trying to run that command. Is this indicative of an issue?

 

fstrim: /mnt/cache: the discard operation is not supported

 

Thanks,

Phil

 

 

It's an indication trim is not supported by one of the ssds or the controller.

OK,

 

So since my last post I've done a few more changes to the array and rooted out the cause of the behaviour if not the root cause of the actual issue.

 

I removed the Sandisk 128GB cache drive from the pool following the instructions in the user manual.

 

After this, the speed of the cache drive is back to expected levels.

 

Therefore I conclude that the problem is arising from:

1) Issues with TRIM which I cannot run/solve using the BR10i controller

2) Issues with the cache pool itself resulting in write issues?

3) An issue with the sandisk 128GB SSD - to be confirmed by adding to the array and running write tests

 

Has anyone else had/seen issues with cache pools and timeouts/speed drops during writes?

hi,

 

yes same here but even worse - copying or moving files to a user share is slow and bumpy to say the least - as copy/move source : smb shares with a windows box are getting reset or even lost till i reset the local nic or worse have to restart the OS / my computer . this behaviour occurs with and without any cache pool or even a single cache disk at my LSI sas2 controller.

 

trim seems not to work as long as the cache disks are connectet to the LSI - i moved a single cache drive to the onboard controller and the trim command is executed correctly will try to pool them again and report on succ/failure

 

cheers ahab