Very slow read access on 4.5.1

jordan · August 9, 2012

This is, as far as I can tell, a problem which has cropped up in the last month or two. The server has been running 4.5.1 for over a year without issue. I recently added a drive (2TB WD20EARS) but the access slowdowns are not limited to this drive. The throughput over the network drops from a typical 30MB/s to 1MB/s or less, with frequent "stalls" and slow access to directories (0.5-1.5 second delays).

Configuration is:

parity Hitachi_HDS5C302_ML0221F308AYND 40°C 1,953,514,552 - 50,576 59,561 0

disk1 WDC_WD10EACS-00Z_WD-WCASJ1189574 41°C 976,761,496 58,283,952 9,330 18 0

disk2 WDC_WD10EACS-00Z_WD-WCASJ1226002 42°C 976,762,552 56,524,440 66,173 23 0

disk3 Hitachi_HDS72101_GTF000PAG6Z6AF 42°C 976,762,552 52,243,348 9,377 35 0

disk4 Hitachi_HDT72101_STF605MS0NMG8K 39°C 976,762,552 49,796,712 129,489 22 0

disk5 Hitachi_HDT72101_STN604MR3X9WPP 40°C 976,762,552 64,196,148 65,934 1,884 0

disk6 WDC_WD10EVCS-63E_WD-WCAU42346275 43°C 976,762,552 78,264,860 71,444 601 0

disk7 ST31000340AS_9QJ05Z8Y 34°C 976,762,552 114,121,848 8,422 250 0

disk8 WDC_WD20EARS-00J_WD-WCAWZ1861514 41°C 1,953,514,552 1,773,233,304 102,308 48,908 0

The syslog doesn't appear to show anything unusual:

Aug 9 05:29:35 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.itl (95) Operation not supported

Aug 9 05:29:35 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 05:29:50 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.xml (95) Operation not supported

Aug 9 05:29:50 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 05:29:50 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.itl (95) Operation not supported

Aug 9 05:29:50 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 05:30:10 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.xml (95) Operation not supported

Aug 9 05:30:10 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 05:30:42 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.itl (95) Operation not supported

Aug 9 05:30:42 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 05:30:53 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.xml (95) Operation not supported

Aug 9 05:30:53 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 07:38:17 Tower1 vsftpd[1912]: connect from 192.168.0.92 (192.168.0.92)

Aug 9 07:38:17 Tower1 vsftpd: Thu Aug 9 07:38:17 2012 [pid 1913] CONNECT: Client "192.168.0.92"

Aug 9 07:38:17 Tower1 vsftpd: Thu Aug 9 07:38:17 2012 [pid 1912] [root] OK LOGIN: Client "192.168.0.92"

Aug 9 07:38:33 Tower1 vsftpd[1915]: connect from 192.168.0.92 (192.168.0.92)

Aug 9 07:38:33 Tower1 vsftpd: Thu Aug 9 07:38:33 2012 [pid 1916] CONNECT: Client "192.168.0.92"

Aug 9 07:38:33 Tower1 vsftpd: Thu Aug 9 07:38:33 2012 [pid 1915] [root] OK LOGIN: Client "192.168.0.92"

Aug 9 07:38:35 Tower1 vsftpd: Thu Aug 9 07:38:35 2012 [pid 1917] [root] OK UPLOAD: Client "192.168.0.92", "/user/work/_TEI/Scan/SKMBT_C36012080909480.pdf", 279293 bytes, 151.26Kbyte/sec

Aug 9 09:41:46 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/4C20524F.tmp (95) Operation not supported

Aug 9 09:41:46 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/School Progression.pub (95) Operation not supported

Aug 9 09:42:13 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/F94C2035.tmp (95) Operation not supported

Aug 9 09:42:13 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/School Progression.pub (95) Operation not supported

Aug 9 09:44:01 Tower1 vsftpd[1922]: connect from 192.168.0.92 (192.168.0.92)

Aug 9 09:44:01 Tower1 vsftpd: Thu Aug 9 09:44:01 2012 [pid 1923] CONNECT: Client "192.168.0.92"

Aug 9 09:44:01 Tower1 vsftpd: Thu Aug 9 09:44:01 2012 [pid 1922] [root] OK LOGIN: Client "192.168.0.92"

Aug 9 09:44:07 Tower1 vsftpd[1925]: connect from 192.168.0.92 (192.168.0.92)

Aug 9 09:44:07 Tower1 vsftpd: Thu Aug 9 09:44:07 2012 [pid 1926] CONNECT: Client "192.168.0.92"

Aug 9 09:44:07 Tower1 vsftpd: Thu Aug 9 09:44:07 2012 [pid 1925] [root] OK LOGIN: Client "192.168.0.92"

Aug 9 09:44:11 Tower1 vsftpd: Thu Aug 9 09:44:11 2012 [pid 1927] [root] OK UPLOAD: Client "192.168.0.92", "/user/work/_TEI/Scan/SKMBT_C36012080911540.pdf", 1691369 bytes, 388.15Kbyte/sec

Aug 9 09:47:43 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/BD85B50B.tmp (95) Operation not supported

Aug 9 09:47:43 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/School Progression.pub (95) Operation not supported

Aug 9 09:50:35 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/C1D5DE51.tmp (95) Operation not supported

Aug 9 09:50:35 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/School Progression.pub (95) Operation not supported

Aug 9 10:45:52 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.itl (95) Operation not supported

Aug 9 10:45:52 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 10:46:14 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.xml (95) Operation not supported

Aug 9 10:46:14 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 10:49:17 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.itl (95) Operation not supported

Aug 9 10:49:17 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 10:49:37 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.xml (95) Operation not supported

Aug 9 10:49:37 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 10:51:17 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.itl (95) Operation not supported

Aug 9 10:51:17 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 10:51:35 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/iTunes Library.xml (95) Operation not supported

Aug 9 10:51:35 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/iTunes - Jordan/Temp File 1.tmp (95) Operation not supported

Aug 9 11:30:30 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/BAE64587.tmp (95) Operation not supported

Aug 9 11:30:30 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/School Progression.pub (95) Operation not supported

Aug 9 11:32:07 Tower1 in.telnetd[1942]: connect from 192.168.0.141 (192.168.0.141)

Aug 9 11:32:10 Tower1 login[1943]: ROOT LOGIN on `pts/0' from `192.168.0.141'

Aug 9 11:32:19 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/79A1D02D.tmp (95) Operation not supported

Aug 9 11:32:19 Tower1 shfs: shfs_setxattr: setxattr: /mnt/disk8/personal/Margaret Beeks 50th/School Progression.pub (95) Operation not supported

Note that the file access which is slow is on all drives, as far as I can tell, not just Disk 8 (a movie is currently "stuttering" in the other room off if, I think , Drive 4.).

A shut-down/hard reboot a week ago fixed the problem, but clearly it is back. I'm not running anything else on the system - no extensions, nothing. Just vanilla 4.5.1. Any suggestions on where else too look for problems? No other network issues seem apparent or have limited bandwidth (and, again, the last time I had issues a reboot of just the unRaid server fixed the symptoms).

jordan · August 9, 2012

Didn't quite get the last sentence in before accidentally hitting something that posted:

Any suggestions on where to look or other places where I might find logs that could help me trace this issue down? I'm not a stranger to the command line, but I'm also not terribly familiar with linux internals so where to find is almost as important as what to look for. ;-)

dgaschk · August 9, 2012

Attach the entire syslog. zip if needed.

jordan · August 10, 2012

Here's the whole log since I noticed the problem and rebooted:

syslog.txt

JonathanM · August 10, 2012

Here's the whole log since I noticed the problem and rebooted:

Probably need a syslog that includes the bad behaviour.

Very slow read access on 4.5.1

Recommended Posts

jordan

Link to comment

jordan

Link to comment

dgaschk

Link to comment

jordan

Link to comment

JonathanM

Link to comment

Archived