errors copying files to SMB share

[skoj]

Apologies for the long post... 

 

I've upgraded from 4.7 to 5.0-rc3 (and then to rc4). This problem appears on both rc3 & rc4 not in 4.7. No addons running, only unmenu is installed but I've disabled it.

 

Since upgrading I've noticed that a large percentage of file copy operations TO unraid via SMB are failing. They're either VERY slow (100-500KB/sec) or they fail outright (windows explorer reports "the specified network name is no longer available" after a few seconds).

 

Could be a network issue but I don't think so. Reading from unraid works perfectly and (when I don't run into this problem) writes work perfectly as well. The problem only affects drives that are 99% full (~30-70GB of 2TB free). I don't always run into the problem on those drives but I never run into it on the drive with 100+ GB free.

 

No errors in the syslog, smart reports are clean, reiserfsck --check doesn't report corruption on any of the disks.

 

One final data point: unraid seems to be writing data to the target file even after the file copy errors out on my windows machine. It is not writing valid data, it is very slow (~100-500KB/sec) but it's definitely writing something to disk. I confirmed (using lsof)that smbd & shfs both have that file open for writing and I can see that blocks are being written to disk by watching the Device Status page on web interface. It eventually finishes but it takes 2 hours for 694MB file.

 

Another thing...unraid seems to be generating far too many read operations when writing the file. See the attached screenshot. There's no way unraid should generate 340k read operations when writing a 694MB file to disk. I'm certain there aren't any other clients reading/writing to unraid & there aren't any plugins running.

 

With 4.7, I maintained drives with as little as 5-10GB free. Performance suffered when writing sometimes but it never errored out like this.

 

Can anyone help? This sucks if I have to keep ~100GB free per disk as it's a management hassle and that adds up to to a cool terabyte with 10 drives!

unraid-slow-file-copy-syslog.zip

[Rajahal]

The problem only affects drives that are 99% full (~30-70GB of 2TB free).

 

I think that may be your answer. Are your data on these drives typically large video files, or lots of small files such as photos, documents, etc? If the latter, then your drive may be heavily fragmented. This means that even though the total amount of free space may be 30 - 70 GB, the actual empty blocks on the drive available for new data to be written to are spread out and very small. This means that to write a 700 MB video file the drive head may have to write little bits of the file all over the place, which significantly slows down the operation.

 

It sounds like you are using the default windows explorer file transfer dialog. I highly recommend you use Teracopy instead. While it may not solve the problem, it will at least give you a bit more information about what is going on.

 

How to solve this? Well, in Windows you would defragment the drive. According to this thread there's a way to do it in unRAID as well, but I personally have not ever tried it. I wouldn't take that action just yet, let's gather some more information first.

 

Other possibilites:

It could be that you are having a network issue that results in very slow writes, but since your reads are normal that seems unlikely. It could also be that some of your disks are becoming unhealthy and not handling the write operations as well. Obtaining and posting SMART reports of the offending drives would be a good idea. If you've been using a cache drive to buffer writes, then the fragmentation theory is most likely not the issue.

[cyrnel]

NIC problems do usually show up more on writes (to array) than reads. It's easier to pump stuff out than receive it. Still, no signs the driver is complaining. Can we see an ifconfig eth0 for comfort?

 

Why is "unraid driver" continually being removed? Crazy number of device inventories. I still can't read a 5.x syslog...

[limetech]

Why is "unraid driver" continually being removed?

FYI - the driver is unloaded/reloaded every time a webGui page is refreshed while the array is Stopped.  This is to detect cases of the user hot plugging drives in and out of the server.

[skoj]

Why is "unraid driver" continually being removed?

FYI - the driver is unloaded/reloaded every time a webGui page is refreshed while the array is Stopped.  This is to detect cases of the user hot plugging drives in and out of the server.

 

Yeah, this syslog was right after upgrading to rc4. I was checking the partition alignment (per the docs) which involved stoppping the array, clicking on each disk on the main page once.

[skoj]

Thanks Rajahal. I also think fragmentation is involved but it just doesn't feel like the impact should be as extreme as this. I've never seen a modern filesystem lose the ability to write files with 30GB+ free. Maybe when you have millions of tiny files but I have max 10k files per drive.

 

I think that may be your answer. Are your data on these drives typically large video files, or lots of small files such as photos, documents, etc? If the latter, then your drive may be heavily fragmented.

Mostly video files + the various jpg & nfo files generated by XBMC when you export your data to individual folders. Roughly ~10k files total with 20% being video files, rest being the xbmc data.

 

It sounds like you are using the default windows explorer file transfer dialog. I highly recommend you use Teracopy instead. While it may not solve the problem, it will at least give you a bit more information about what is going on.

I'm actually using  an explorer replacement tool called Directory Opus but I'll give TeraCopy a try too.

 

Obtaining and posting SMART reports of the offending drives would be a good idea. If you've been using a cache drive to buffer writes, then the fragmentation theory is most likely not the issue.

I checked the SMART reports, they look clean to me but I might have missed something so I'll post them tonight. Also, no cache drive is installed.