November 14, 201312 yr I'm having a problem with the mover script in my system. I had been running for years without a cache drive and everything was working fine. 10 protected discs, Plex Media Server, unMENU, Time Machine, AFP and SMB. I'm currently running 5.0-Final under a Pro license but have followed the 5.0 beta program and haven't had any troubles. I finally was able to reconfigure my hardware with a couple of new drives and free up an old one for use as a cache disk. I mainly wanted to use the cache to speed up file transfer and to hold the database info for Plex. After the upgrade all hell has broken loose. I was having problems with hanging on boot (fixed with a new USB Drive), freezes, kernel panics, crashes, all sorts of fun. I had to repair the file system on one of my data disks that didn't quite make it through the upgrade cleanly, but that's now all working again. The system seems to have settled down a little after, new permission scripts, removal of some .* apple cruft at the top level of user shares, and a completely clean reinstall on the new USB drive. The only things I have running now are the stock 5.0-Final and Plex Media Server. I've done an overnight Memtest and it has come out clean. The main problem I have right now is that when I get the mover script actually working, it seems to remove all traces of whatever file it was trying to move in the first place. It's no longer on the cache and hasn't been moved to the array either. I have disabled AFP on all shares except for my Time Machine share (which I currently have disabled and isn't using the cache...one problem at a time) as I read somewhere that sometimes AFP doesn't play well with a cache drive. I'm at the "I don't know where to go next" point. I'm normally able to teach myself whatever I need to know to fix a problem through enough googling and trial and error. I'm not a very proficient linux user but I don't mind hitting the command line if I know the commands to run. I have attached a large dump of my current system log (activity over the past 2 days). I'm hoping that a more experienced user can scan through the log and give me an idea as to what might actually be going on. I've tried but I can't quite seem to distill the root cause. There seems to be a bunch of entries based on memory paging when the mover runs, but there also seems to be a lot of chatter coming from Plex as well. I don't know if the two are related or not. Any help or insight would be greatly appreciated. I would hate to have to scrap the idea of a cache drive completely and just go back to a standard array, there seems to be so many great upsides to having one. Many Thanks. syslog.txt
November 14, 201312 yr It is likely an incompatible add-on. Have you updated all of the add-ons? See my sig to disable all add-ons for testing.
November 14, 201312 yr Author I have disabled Plex Media Centre, the only plugin that was running. Tried to reboot and had the web interface crash trying to unmount user shares. Had to do a hard reboot. Once restarted I copied a file to the cache drive through my Media share and invoked the mover from the share settings page. The mover did not complete and there was also a bunch of messages from the unRAID kernel ending in "unRAID kernel: Fixing recursive fault but reboot is needed!". I have seen this before as well. Not sure what it means. New syslog attached. The cache drive and Parity are connected to a new PCI-E sata expansion card. Would this have anything to do with it? syslog2.txt
November 14, 201312 yr Author Update: I have tried a new SATA cable. No luck. When I engaged the mover from the web interface the mover would crash and so would the web interface. Did a hard reboot to get it back. On the upside (I guess) the test file was still on the cache drive and hadn't disappeared like before. I then tried starting the mover from a telnet session (also so that I could capture console output). Same problem, doesn't go. However this time the web interface stayed responsive. I have pasted the output of the console below: mover started skipping Apps/ moving Media/ ./Media/TV Shows/Marvel's Agents of SHIELD/Season 1/Marvels.Agents.of.S.H.I.E.L.D.S01E01.720p.HDTV.X264-DIMENSION.mkv *** glibc detected *** rsync: free(): invalid next size (normal): 0x080d2488 *** ======= Backtrace: ========= /lib/libc.so.6(+0x705aa)[0x400a45aa] /lib/libc.so.6(+0x73503)[0x400a7503] /lib/libc.so.6(cfree+0x70)[0x400aa6b0] rsync[0x807cd74] rsync[0x807de60] rsync[0x804f3aa] rsync[0x8050b5f] rsync[0x8051e56] rsync[0x8065825] rsync[0x80666ac] /lib/libc.so.6(__libc_start_main+0xe6)[0x4004ab86] rsync[0x804aad1] ======= Memory map: ======== 08048000-0809d000 r-xp 00000000 00:01 2156 /usr/bin/rsync 0809d000-080a1000 rwxp 00054000 00:01 2156 /usr/bin/rsync 080a1000-080f2000 rwxp 00000000 00:00 0 [heap] 40000000-4001d000 r-xp 00000000 00:01 552 /lib/ld-2.11.1.so 4001d000-4001e000 r-xp 0001d000 00:01 552 /lib/ld-2.11.1.so 4001e000-4001f000 rwxp 0001e000 00:01 552 /lib/ld-2.11.1.so 4001f000-40020000 r-xp 00000000 00:00 0 [vdso] 40020000-40021000 rwxp 00000000 00:00 0 40026000-4002c000 r-xp 00000000 00:01 881 /lib/libacl.so.1.1.0 4002c000-4002d000 rwxp 00005000 00:01 881 /lib/libacl.so.1.1.0 4002d000-40033000 r-xp 00000000 00:01 828 /lib/libpopt.so.0.0.0 40033000-40034000 rwxp 00006000 00:01 828 /lib/libpopt.so.0.0.0 40034000-40190000 r-xp 00000000 00:01 878 /lib/libc-2.11.1.so 40190000-40191000 ---p 0015c000 00:01 878 /lib/libc-2.11.1.so 40191000-40193000 r-xp 0015c000 00:01 878 /lib/libc-2.11.1.so 40193000-40194000 rwxp 0015e000 00:01 878 /lib/libc-2.11.1.so 40194000-40197000 rwxp 00000000 00:00 0 40197000-4019b000 r-xp 00000000 00:01 544 /lib/libattr.so.1.1.0 4019b000-4019c000 rwxp 00003000 00:01 544 /lib/libattr.so.1.1.0 4019c000-4019e000 rwxp 00000000 00:00 0 4019e000-401dd000 r-xp 00000000 00:01 2699 /usr/lib/locale/en_US.utf8/LC_CTYPE 401dd000-4023f000 rwxp 00000000 00:00 0 4023f000-4025b000 r-xp 00000000 00:01 2633 /usr/lib/libgcc_s.so.1 4025b000-4025c000 rwxp 0001b000 00:01 2633 /usr/lib/libgcc_s.so.1 40300000-40321000 rwxp 00000000 00:00 0 40321000-40400000 ---p 00000000 00:00 0 73750000-bfbe7000 rw-p 00000000 00:00 0 [stack] find: `rsync' terminated by signal 6 rsync: writefd_unbuffered failed to write 79 bytes to socket [Receiver]: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(1530) [Receiver=3.0.7] ./Media/TV Shows/Marvel's Agents of SHIELD/Season 1/._Marvels.Agents.of.S.H.I.E.L.D.S01E01.720p.HDTV.X264-DIMENSION.mkv *** glibc detected *** rsync: free(): invalid next size (normal): 0x080d2488 *** ======= Backtrace: ========= /lib/libc.so.6(+0x705aa)[0x400a45aa] /lib/libc.so.6(+0x73503)[0x400a7503] /lib/libc.so.6(cfree+0x70)[0x400aa6b0] rsync[0x807cd74] rsync[0x807de60] rsync[0x804f3aa] rsync[0x8050b5f] rsync[0x8051e56] rsync[0x8065825] rsync[0x80666ac] /lib/libc.so.6(__libc_start_main+0xe6)[0x4004ab86] rsync[0x804aad1] ======= Memory map: ======== 08048000-0809d000 r-xp 00000000 00:01 2156 /usr/bin/rsync 0809d000-080a1000 rwxp 00054000 00:01 2156 /usr/bin/rsync 080a1000-080f2000 rwxp 00000000 00:00 0 [heap] 40000000-4001d000 r-xp 00000000 00:01 552 /lib/ld-2.11.1.so 4001d000-4001e000 r-xp 0001d000 00:01 552 /lib/ld-2.11.1.so 4001e000-4001f000 rwxp 0001e000 00:01 552 /lib/ld-2.11.1.so 4002d000-40033000 r-xp 00000000 00:01 828 /lib/libpopt.so.0.0.0 40033000-40034000 rwxp 00006000 00:01 828 /lib/libpopt.so.0.0.0 40034000-40190000 r-xp 00000000 00:01 878 /lib/libc-2.11.1.so 40190000-40191000 ---p 0015c000 00:01 878 /lib/libc-2.11.1.so 40191000-40193000 r-xp 0015c000 00:01 878 /lib/libc-2.11.1.so 40193000-40194000 rwxp 0015e000 00:01 878 /lib/libc-2.11.1.so 40194000-40197000 rwxp 00000000 00:00 0 40197000-4019b000 r-xp 00000000 00:01 544 /lib/libattr.so.1.1.0 4019b000-4019c000 rwxp 00003000 00:01 544 /lib/libattr.so.1.1.0 4019c000-4019e000 rwxp 00000000 00:00 0 4019e000-401dd000 r-xp 00000000 00:01 2699 /usr/lib/locale/en_US.utf8/LC_CTYPE 401dd000-4023f000 rwxp 00000000 00:00 0 4023f000-4025b000 r-xp 00000000 00:01 2633 /usr/lib/libgcc_s.so.1 4025b000-4025c000 rwxp 0001b000 00:01 2633 /usr/lib/libgcc_s.so.1 40300000-40321000 rwxp 00000000 00:00 0 40321000-40400000 ---p 00000000 00:00 0 73750000-bfe82000 rw-p 00000000 00:00 0 [stack] find: `rsync' terminated by signal 6 rsync: writefd_unbuffered failed to write 79 bytes to socket [Receiver]: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(1530) [Receiver=3.0.7] ./Media/TV Shows/Marvel's Agents of SHIELD/Season 1 *** glibc detected *** rsync: free(): invalid next size (normal): 0x080d2440 *** ======= Backtrace: ========= /lib/libc.so.6(+0x705aa)[0x400a45aa] /lib/libc.so.6(+0x73503)[0x400a7503] /lib/libc.so.6(cfree+0x70)[0x400aa6b0] rsync[0x807cd74] rsync[0x807de60] rsync[0x804f3aa] rsync[0x8050b5f] rsync[0x8051e56] rsync[0x8065825] rsync[0x80666ac] /lib/libc.so.6(__libc_start_main+0xe6)[0x4004ab86] rsync[0x804aad1] ======= Memory map: ======== 08048000-0809d000 r-xp 00000000 00:01 2156 /usr/bin/rsync 0809d000-080a1000 rwxp 00054000 00:01 2156 /usr/bin/rsync 080a1000-080f2000 rwxp 00000000 00:00 0 [heap] 40000000-4001d000 r-xp 00000000 00:01 552 /lib/ld-2.11.1.so 4001d000-4001e000 r-xp 0001d000 00:01 552 /lib/ld-2.11.1.so 4001e000-4001f000 rwxp 0001e000 00:01 552 /lib/ld-2.11.1.so 4001f000-40020000 r-xp 00000000 00:00 0 [vdso] 40020000-40021000 rwxp 00000000 00:00 0 40026000-4002c000 r-xp 00000000 00:01 881 /lib/libacl.so.1.1.0 4002c000-4002d000 rwxp 00005000 00:01 881 /lib/libacl.so.1.1.0 4002d000-40033000 r-xp 00000000 00:01 828 /lib/libpopt.so.0.0.0 40033000-40034000 rwxp 00006000 00:01 828 /lib/libpopt.so.0.0.0 40034000-40190000 r-xp 00000000 00:01 878 /lib/libc-2.11.1.so 40190000-40191000 ---p 0015c000 00:01 878 /lib/libc-2.11.1.so 40191000-40193000 r-xp 0015c000 00:01 878 /lib/libc-2.11.1.so 40193000-40194000 rwxp 0015e000 00:01 878 /lib/libc-2.11.1.so 40194000-40197000 rwxp 00000000 00:00 0 40197000-4019b000 r-xp 00000000 00:01 544 /lib/libattr.so.1.1.0 4019b000-4019c000 rwxp 00003000 00:01 544 /lib/libattr.so.1.1.0 4019c000-4019e000 rwxp 00000000 00:00 0 4019e000-401dd000 r-xp 00000000 00:01 2699 /usr/lib/locale/en_US.utf8/LC_CTYPE 401dd000-4023f000 rwxp 00000000 00:00 0 4023f000-4025b000 r-xp 00000000 00:01 2633 /usr/lib/libgcc_s.so.1 4025b000-4025c000 rwxp 0001b000 00:01 2633 /usr/lib/libgcc_s.so.1 40300000-40321000 rwxp 00000000 00:00 0 40321000-40400000 ---p 00000000 00:00 0 73750000-bf97c000 rw-p 00000000 00:00 0 [stack] Message from syslogd@unRAID at Thu Nov 14 18:31:12 2013 ... unRAID kernel: Process rsync (pid: 1623, ti=f77ac000 task=f13210e0 task.ti=f77ac000) Message from syslogd@unRAID at Thu Nov 14 18:31:12 2013 ... unRAID kernel: Call Trace: Message from syslogd@unRAID at Thu Nov 14 18:31:12 2013 ... unRAID kernel: Code: 00 00 8d b0 00 00 00 40 c1 ee 0c c1 e6 05 03 35 00 0d 61 c1 8b 06 66 85 c0 79 03 8b 76 1c 8b 06 84 c0 78 20 66 f7 06 00 c0 75 04 <0f> 0b eb fe 8b 06 31 d2 f6 c4 40 74 03 8b 56 38 89 f0 e8 3d de Message from syslogd@unRAID at Thu Nov 14 18:31:12 2013 ... unRAID kernel: EIP: [<c1093683>] kfree+0x3d/0xbe SS:ESP 0068:f77ad8e4 Message from syslogd@unRAID at Thu Nov 14 18:31:12 2013 ... unRAID kernel: Stack:
November 15, 201312 yr Author That's great! Thank you. Went through and processed all of my array disks and attempted a move. It crashed, but after I restarted, removed the data from the cache drive, and re-added the files it all works well. No crashing or problems unmounting shares when trying to stop the array. Who knew such a thing could cause so much havoc. I've even re-enabled Plex and had no problems with the mover script. Thanks again for your help.
Archived
This topic is now archived and is closed to further replies.