Mover crashing server

Rich · August 18, 2012

Having completely guessed it might be permissions related i ran the ‘new permissions’ utility, which had no visible effect. Then i decided to test my other ‘shares’ and to my surprise, mover worked perfectly with all of them.

So the system crash issue is limited to my 'media' share and only happens when there is something to ‘move’ to the share from the cache drive. All my other shares ‘move’ as they should.

Syslogs listed below (scrolling needed with the second one). I would really appreciate some help with this one, i'm completely out of ideas now and definatley out of my depth $:-\$

Syslogs...

Test 'move' with 'Files and Programs' share - Successful

Aug 18 17:42:54 unRAID logger: mover started
Aug 18 17:42:54 unRAID logger: moving Files and Programs/
Aug 18 17:42:54 unRAID logger: ./Files and Programs/._starwars force unleashed.mov
Aug 18 17:42:54 unRAID logger: .d..t...... ./
Aug 18 17:42:54 unRAID logger: .d..t...... Files and Programs/
Aug 18 17:42:54 unRAID logger: >f+++++++++ Files and Programs/._starwars force unleashed.mov
Aug 18 17:42:54 unRAID logger: ./Files and Programs/starwars force unleashed.mov
Aug 18 17:42:54 unRAID logger: >f+++++++++ Files and Programs/starwars force unleashed.mov
Aug 18 17:42:54 unRAID shfs/user: duplicate object: /mnt/disk4/Files and Programs/starwars force unleashed.mov
Aug 18 17:42:54 unRAID shfs/user: duplicate object: /mnt/disk4/Files and Programs/starwars force unleashed.mov
Aug 18 17:42:54 unRAID logger: ./Files and Programs/
Aug 18 17:42:55 unRAID logger: skipping PLEX/
Aug 18 17:42:55 unRAID logger: mover finished

Test 'move' with 'Media' share - Unsuccessful (entire system crashed)

Aug 18 17:36:10 unRAID logger: mover started
Aug 18 17:36:10 unRAID logger: moving Media/
Aug 18 17:36:10 unRAID logger: ./Media/._starwars force unleashed.mov
Aug 18 17:36:10 unRAID logger: *** glibc detected *** rsync: free(): invalid next size (normal): 0x080cbfa0 ***
Aug 18 17:36:10 unRAID logger: ======= Backtrace: =========
Aug 18 17:36:10 unRAID logger: /lib/libc.so.6(+0x705aa)[0xb75df5aa]
Aug 18 17:36:10 unRAID logger: /lib/libc.so.6(+0x73503)[0xb75e2503]
Aug 18 17:36:10 unRAID logger: /lib/libc.so.6(cfree+0x70)[0xb75e56b0]
Aug 18 17:36:10 unRAID logger: rsync[0x807cd74]
Aug 18 17:36:10 unRAID logger: rsync[0x807de60]
Aug 18 17:36:10 unRAID logger: rsync[0x804f3aa]
Aug 18 17:36:10 unRAID logger: rsync[0x8050bbe]
Aug 18 17:36:10 unRAID logger: rsync[0x8051e56]
Aug 18 17:36:10 unRAID logger: rsync[0x8065825]
Aug 18 17:36:10 unRAID logger: rsync[0x80666ac]
Aug 18 17:36:10 unRAID logger: /lib/libc.so.6(__libc_start_main+0xe6)[0xb7585b86]
Aug 18 17:36:10 unRAID logger: rsync[0x804aad1]
Aug 18 17:36:10 unRAID logger: ======= Memory map: ========
Aug 18 17:36:10 unRAID logger: 08048000-0809d000 r-xp 00000000 00:01 2120       /usr/bin/rsync
Aug 18 17:36:10 unRAID logger: 0809d000-080a1000 rw-p 00054000 00:01 2120       /usr/bin/rsync
Aug 18 17:36:10 unRAID logger: 080a1000-080f3000 rw-p 00000000 00:00 0          [heap]
Aug 18 17:36:10 unRAID logger: b7400000-b7421000 rw-p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b7421000-b7500000 ---p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b750a000-b7526000 r-xp 00000000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 18 17:36:10 unRAID logger: b7526000-b7527000 rw-p 0001b000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 18 17:36:10 unRAID logger: b7527000-b756a000 rw-p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b756a000-b756e000 r-xp 00000000 00:01 545        /lib/libattr.so.1.1.0
Aug 18 17:36:10 unRAID logger: b756e000-b756f000 rw-p 00003000 00:01 545        /lib/libattr.so.1.1.0
Aug 18 17:36:10 unRAID logger: b756f000-b76cb000 r-xp 00000000 00:01 842        /lib/libc-2.11.1.so
Aug 18 17:36:10 unRAID logger: b76cb000-b76cc000 ---p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 18 17:36:10 unRAID logger: b76cc000-b76ce000 r--p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 18 17:36:10 unRAID logger: b76ce000-b76cf000 rw-p 0015e000 00:01 842        /lib/libc-2.11.1.so
Aug 18 17:36:10 unRAID logger: b76cf000-b76d2000 rw-p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b76d2000-b76d8000 r-xp 00000000 00:01 792        /lib/libpopt.so.0.0.0
Aug 18 17:36:10 unRAID logger: b76d8000-b76d9000 rw-p 00006000 00:01 792        /lib/libpopt.so.0.0.0
Aug 18 17:36:10 unRAID logger: b76d9000-b76df000 r-xp 00000000 00:01 845        /lib/libacl.so.1.1.0
Aug 18 17:36:10 unRAID logger: b76df000-b76e0000 rw-p 00005000 00:01 845        /lib/libacl.so.1.1.0
Aug 18 17:36:10 unRAID logger: b76e5000-b76e6000 rw-p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b76e6000-b76e7000 r-xp 00000000 00:00 0          [vdso]
Aug 18 17:36:10 unRAID logger: b76e7000-b7704000 r-xp 00000000 00:01 553        /lib/ld-2.11.1.so
Aug 18 17:36:10 unRAID logger: b7704000-b7705000 r--p 0001d000 00:01 553        /lib/ld-2.11.1.so
Aug 18 17:36:10 unRAID logger: b7705000-b7706000 rw-p 0001e000 00:01 553        /lib/ld-2.11.1.so
Aug 18 17:36:10 unRAID logger: bfe7e000-bfe9f000 rw-p 00000000 00:00 0          [stack]
Aug 18 17:36:10 unRAID logger: find: `rsync' terminated by signal 6
Aug 18 17:36:10 unRAID logger: rsync: writefd_unbuffered failed to write 79 bytes to socket [Receiver]: Broken pipe (32)
Aug 18 17:36:10 unRAID logger: rsync error: error in rsync protocol data stream (code 12) at io.c(1530) [Receiver=3.0.7]
Aug 18 17:36:10 unRAID logger: ./Media/starwars force unleashed.mov
Aug 18 17:36:10 unRAID logger: *** glibc detected *** rsync: free(): invalid next size (normal): 0x080cbfa0 ***
Aug 18 17:36:10 unRAID logger: ======= Backtrace: =========
Aug 18 17:36:10 unRAID logger: /lib/libc.so.6(+0x705aa)[0xb76725aa]
Aug 18 17:36:10 unRAID logger: /lib/libc.so.6(+0x73503)[0xb7675503]
Aug 18 17:36:10 unRAID logger: /lib/libc.so.6(cfree+0x70)[0xb76786b0]
Aug 18 17:36:10 unRAID logger: rsync[0x807cd74]
Aug 18 17:36:10 unRAID logger: rsync[0x807de60]
Aug 18 17:36:10 unRAID logger: rsync[0x804f3aa]
Aug 18 17:36:10 unRAID logger: rsync[0x8050bbe]
Aug 18 17:36:10 unRAID logger: rsync[0x8051e56]
Aug 18 17:36:10 unRAID logger: rsync[0x8065825]
Aug 18 17:36:10 unRAID logger: rsync[0x80666ac]
Aug 18 17:36:10 unRAID logger: /lib/libc.so.6(__libc_start_main+0xe6)[0xb7618b86]
Aug 18 17:36:10 unRAID logger: rsync[0x804aad1]
Aug 18 17:36:10 unRAID logger: ======= Memory map: ========
Aug 18 17:36:10 unRAID logger: 08048000-0809d000 r-xp 00000000 00:01 2120       /usr/bin/rsync
Aug 18 17:36:10 unRAID logger: 0809d000-080a1000 rw-p 00054000 00:01 2120       /usr/bin/rsync
Aug 18 17:36:10 unRAID logger: 080a1000-080f3000 rw-p 00000000 00:00 0          [heap]
Aug 18 17:36:10 unRAID logger: b7400000-b7421000 rw-p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b7421000-b7500000 ---p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b759d000-b75b9000 r-xp 00000000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 18 17:36:10 unRAID logger: b75b9000-b75ba000 rw-p 0001b000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 18 17:36:10 unRAID logger: b75ba000-b75fd000 rw-p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b75fd000-b7601000 r-xp 00000000 00:01 545        /lib/libattr.so.1.1.0
Aug 18 17:36:10 unRAID logger: b7601000-b7602000 rw-p 00003000 00:01 545        /lib/libattr.so.1.1.0
Aug 18 17:36:10 unRAID logger: b7602000-b775e000 r-xp 00000000 00:01 842        /lib/libc-2.11.1.so
Aug 18 17:36:10 unRAID logger: b775e000-b775f000 ---p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 18 17:36:10 unRAID logger: b775f000-b7761000 r--p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 18 17:36:10 unRAID logger: b7761000-b7762000 rw-p 0015e000 00:01 842        /lib/libc-2.11.1.so
Aug 18 17:36:10 unRAID logger: b7762000-b7765000 rw-p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b7765000-b776b000 r-xp 00000000 00:01 792        /lib/libpopt.so.0.0.0
Aug 18 17:36:10 unRAID logger: b776b000-b776c000 rw-p 00006000 00:01 792        /lib/libpopt.so.0.0.0
Aug 18 17:36:10 unRAID logger: b776c000-b7772000 r-xp 00000000 00:01 845        /lib/libacl.so.1.1.0
Aug 18 17:36:10 unRAID logger: b7772000-b7773000 rw-p 00005000 00:01 845        /lib/libacl.so.1.1.0
Aug 18 17:36:10 unRAID logger: b7778000-b7779000 rw-p 00000000 00:00 0 
Aug 18 17:36:10 unRAID logger: b7779000-b777a000 r-xp 00000000 00:00 0          [vdso]
Aug 18 17:36:10 unRAID logger: b777a000-b7797000 r-xp 00000000 00:01 553        /lib/ld-2.11.1.so
Aug 18 17:36:10 unRAID logger: b7797000-b7798000 r--p 0001d000 00:01 553        /lib/ld-2.11.1.so
Aug 18 17:36:10 unRAID logger: b7798000-b7799000 rw-p 0001e000 00:01 553        /lib/ld-2.11.1.so
Aug 18 17:36:10 unRAID logger: bff64000-bff85000 rw-p 00000000 00:00 0          [stack]
Aug 18 17:36:10 unRAID logger: find: `rsync' terminated by signal 6
Aug 18 17:36:10 unRAID logger: rsync: writefd_unbuffered failed to write 79 bytes to socket [Receiver]: Broken pipe (32)
Aug 18 17:36:10 unRAID logger: rsync error: error in rsync protocol data stream (code 12) at io.c(1530) [Receiver=3.0.7]
Aug 18 17:36:10 unRAID logger: ./Media/

mr-hexen · August 19, 2012

what add-ons do you have loaded??

Rich · August 19, 2012

Currently i have sab, sickbeard, couchpotato and plex running, but the problem happens regardless... i've freshly installed unraid with no extras and still the same problem, so put everything back on.

dgaschk · August 20, 2012

I suggest you debug this with a clean install. Do you have a spare flash to run the free version?

Rich · August 21, 2012

I do yes.

I have already tried a clean install of unraid though and had the same problem, i'll give it another go though. Also, if i use the free version my shares won't be complete because i have too many disks, will that be a problem?

dgaschk · August 21, 2012

Never mind. The free version doesn't support a cache drive. Backup the entire contents of the flash drive and then format and do a clean install. Keep the clean install until this is resolved. You can move back and forth if needed by backing up the flash and replacing the content with the alternate backup.

JonathanM · August 21, 2012

I haven't been fully keeping up, so forgive me if I'm rehashing something already covered, but have you ruled out a problem on the cache drive itself? I'd copy any cache only files onto a temporary spot on the array, run a preclear cycle after removing the cache drive from the array, reassign it and let unraid repartition and format it.

Rich · August 21, 2012

Never mind. The free version doesn't support a cache drive. Backup the entire contents of the flash drive and then format and do a clean install. Keep the clean install until this is resolved. You can move back and forth if needed by backing up the flash and replacing the content with the alternate backup.

Ok, i'll make sure i test any changes i make on a clean install. Unfortunately i cant keep a clean copy of unRAID running all the time as the machine is the primary media server for the house and in constant use during the day.

I haven't been fully keeping up, so forgive me if I'm rehashing something already covered, but have you ruled out a problem on the cache drive itself? I'd copy any cache only files onto a temporary spot on the array, run a preclear cycle after removing the cache drive from the array, reassign it and let unraid repartition and format it.

No, I haven't ruled it out (i'm very much an unRAID user only and not a programer, so i don't have much of a clue when it comes to troubleshooting $:-\$ ). The only reason i wouldn't think it was the cache drive is because the two other shares work perfectly with mover, its just this one (Media) share. Does that not illuminate the cache drive, or could it still be causing the problem?

JonathanM · August 21, 2012

The only reason i wouldn't think it was the cache drive is because the two other shares work perfectly with mover, its just this one (Media) share. Does that not illuminate the cache drive, or could it still be causing the problem?

Since the cache drive is involved in the process, it must be suspect until proven otherwise. Do you keep anything permanently on the cache drive, or is it empty after the mover runs when things are working properly?

Rich · August 21, 2012

There's a cache only share for PLEX and SAB and a .tmp folder.

I'll transfer them all off, pre clear as advised, then report back

Rich · August 22, 2012

Unfortunately the preclear and format didn't work, same problem $:-\$

I tried three different times to invoke the mover (with the Media share) and got a system crash all thee times (mover still works as it should with my other shares).

The three syslogs all look slightly different tho (attached) and the second one is huge, although it doesn't look like it makes any sense?

syslog 1

Aug 22 12:50:08 unRAID logger: mover started
Aug 22 12:50:08 unRAID logger: moving Media/
Aug 22 12:50:08 unRAID logger: ./Media/photo1.jpg
Aug 22 12:50:08 unRAID logger: *** glibc detected *** rsync: free(): invalid next size (normal): 0x080cbf88 ***

syslog 2

https://dl.dropbox.com/u/26618776/Syslog2.rtf

syslog 3

Aug 22 13:22:53 unRAID logger: mover started
Aug 22 13:22:53 unRAID logger: moving Media/
Aug 22 13:22:53 unRAID logger: ./Media/TEST.MOV
Aug 22 13:22:53 unRAID logger: *** glibc detected *** rsync: free(): invalid next size (normal): 0x080cbf88 ***
Aug 22 13:22:53 unRAID logger: ======= Backtrace: =========
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(+0x705aa)[0xb764e5aa]
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(+0x73503)[0xb7651503]
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(cfree+0x70)[0xb76546b0]
Aug 22 13:22:53 unRAID logger: rsync[0x807cd74]
Aug 22 13:22:53 unRAID logger: rsync[0x807de60]
Aug 22 13:22:53 unRAID logger: rsync[0x804f3aa]
Aug 22 13:22:53 unRAID logger: rsync[0x8050bbe]
Aug 22 13:22:53 unRAID logger: rsync[0x8051e56]
Aug 22 13:22:53 unRAID logger: rsync[0x8065825]
Aug 22 13:22:53 unRAID logger: rsync[0x80666ac]
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(__libc_start_main+0xe6)[0xb75f4b86]
Aug 22 13:22:53 unRAID logger: rsync[0x804aad1]
Aug 22 13:22:53 unRAID logger: ======= Memory map: ========
Aug 22 13:22:53 unRAID logger: 08048000-0809d000 r-xp 00000000 00:01 2120       /usr/bin/rsync
Aug 22 13:22:53 unRAID logger: 0809d000-080a1000 rw-p 00054000 00:01 2120       /usr/bin/rsync
Aug 22 13:22:53 unRAID logger: 080a1000-080f3000 rw-p 00000000 00:00 0          [heap]
Aug 22 13:22:53 unRAID logger: b7400000-b7421000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b7421000-b7500000 ---p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b7579000-b7595000 r-xp 00000000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 22 13:22:53 unRAID logger: b7595000-b7596000 rw-p 0001b000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 22 13:22:53 unRAID logger: b7596000-b75d9000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b75d9000-b75dd000 r-xp 00000000 00:01 545        /lib/libattr.so.1.1.0
Aug 22 13:22:53 unRAID logger: b75dd000-b75de000 rw-p 00003000 00:01 545        /lib/libattr.so.1.1.0
Aug 22 13:22:53 unRAID logger: b75de000-b773a000 r-xp 00000000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b773a000-b773b000 ---p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b773b000-b773d000 r--p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b773d000-b773e000 rw-p 0015e000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b773e000-b7741000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b7741000-b7747000 r-xp 00000000 00:01 792        /lib/libpopt.so.0.0.0
Aug 22 13:22:53 unRAID logger: b7747000-b7748000 rw-p 00006000 00:01 792        /lib/libpopt.so.0.0.0
Aug 22 13:22:53 unRAID logger: b7748000-b774e000 r-xp 00000000 00:01 845        /lib/libacl.so.1.1.0
Aug 22 13:22:53 unRAID logger: b774e000-b774f000 rw-p 00005000 00:01 845        /lib/libacl.so.1.1.0
Aug 22 13:22:53 unRAID logger: b7754000-b7755000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b7755000-b7756000 r-xp 00000000 00:00 0          [vdso]
Aug 22 13:22:53 unRAID logger: b7756000-b7773000 r-xp 00000000 00:01 553        /lib/ld-2.11.1.so
Aug 22 13:22:53 unRAID logger: b7773000-b7774000 r--p 0001d000 00:01 553        /lib/ld-2.11.1.so
Aug 22 13:22:53 unRAID logger: b7774000-b7775000 rw-p 0001e000 00:01 553        /lib/ld-2.11.1.so
Aug 22 13:22:53 unRAID logger: bff9e000-bffbf000 rw-p 00000000 00:00 0          [stack]
Aug 22 13:22:53 unRAID logger: find: `rsync' terminated by signal 6
Aug 22 13:22:53 unRAID logger: rsync: writefd_unbuffered failed to write 79 bytes to socket [Receiver]: Broken pipe (32)
Aug 22 13:22:53 unRAID logger: rsync error: error in rsync protocol data stream (code 12) at io.c(1530) [Receiver=3.0.7]
Aug 22 13:22:53 unRAID kernel: BUG: unable to handle kernel paging request at 737500c9
Aug 22 13:22:53 unRAID kernel: IP: [<c121e878>] scsi_device_get+0x20/0x45
Aug 22 13:22:53 unRAID kernel: *pdpt = 0000000032b0d001 *pde = 0000000000000000 
Aug 22 13:22:53 unRAID kernel: Oops: 0000 [#1] SMP 
Aug 22 13:22:53 unRAID kernel: Modules linked in: md_mod xor asus_atk0110 hwmon mvsas libsas scsi_transport_sas sata_sil24 ata_piix atl1e piix [last unloaded: md_mod]
Aug 22 13:22:53 unRAID kernel: 
Aug 22 13:22:53 unRAID kernel: Pid: 6149, comm: smartctl Not tainted 3.0.31-unRAID #1 System manufacturer System Product Name/P5QPL-AM
Aug 22 13:22:53 unRAID kernel: EIP: 0060:[<c121e878>] EFLAGS: 00210286 CPU: 0
Aug 22 13:22:53 unRAID kernel: EIP is at scsi_device_get+0x20/0x45
Aug 22 13:22:53 unRAID kernel: EAX: 73750069 EBX: f2913c00 ECX: c12292d1 EDX: f2913cb8
Aug 22 13:22:53 unRAID kernel: ESI: fffffffa EDI: f2ea6e00 EBP: f1397dac ESP: f1397da8
Aug 22 13:22:53 unRAID kernel:  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Aug 22 13:22:53 unRAID kernel: Process smartctl (pid: 6149, ti=f1396000 task=f11f7020 task.ti=f1396000)
Aug 22 13:22:53 unRAID kernel: Stack:
Aug 22 13:22:53 unRAID kernel:  f3108400 f1397db8 c12285f9 f3108600 f1397dd4 c12292f8 0000005d f2b3c1f0
Aug 22 13:22:53 unRAID kernel:  f2ea6e00 00000000 f3108600 f1397e00 c10a18a5 00000000 00000000 0000005d
Aug 22 13:22:53 unRAID kernel:  f2ea6e10 f1397e38 00000000 00000000 f2ea6e00 f1bf20c0 f1397e24 c10a19fc
Aug 22 13:22:53 unRAID kernel: Call Trace:
Aug 22 13:22:53 unRAID kernel:  [<c12285f9>] __scsi_disk_get+0x16/0x2b
Aug 22 13:22:53 unRAID kernel:  [<c12292f8>] sd_open+0x27/0x127
Aug 22 13:22:53 unRAID kernel:  [<c10a18a5>] __blkdev_get+0x202/0x2db
Aug 22 13:22:53 unRAID kernel:  [<c10a19fc>] blkdev_get+0x7e/0x149
Aug 22 13:22:53 unRAID kernel:  [<c10a1ac7>] ? blkdev_get+0x149/0x149
Aug 22 13:22:53 unRAID kernel:  [<c10a1b25>] blkdev_open+0x5e/0x65
Aug 22 13:22:53 unRAID kernel:  [<c107ef5b>] __dentry_open+0x11a/0x1f7
Aug 22 13:22:53 unRAID kernel:  [<c107f0cc>] nameidata_to_filp+0x45/0x53
Aug 22 13:22:53 unRAID kernel:  [<c10899ae>] do_last+0x4ee/0x5ba
Aug 22 13:22:53 unRAID kernel:  [<c108a64b>] path_openat+0x95/0x29e
Aug 22 13:22:53 unRAID kernel:  [<c106dd07>] ? handle_pte_fault+0x131/0x20f
Aug 22 13:22:53 unRAID kernel:  [<c108a8fe>] do_filp_open+0x21/0x5f
Aug 22 13:22:53 unRAID kernel:  [<c1089b0c>] ? getname_flags+0x21/0xbe
Aug 22 13:22:53 unRAID kernel:  [<c107ec5e>] do_sys_open+0xf6/0x174
Aug 22 13:22:53 unRAID kernel:  [<c107ed1e>] sys_open+0x1e/0x26
Aug 22 13:22:53 unRAID kernel:  [<c130ed05>] syscall_call+0x7/0xb
Aug 22 13:22:53 unRAID kernel: Code: 00 00 e8 ff 6e fe ff 5b 5e 5d c3 55 89 e5 53 89 c3 83 b8 54 03 00 00 04 74 2a 8d 80 b8 00 00 00 e8 bc 6e fe ff 85 c0 74 1b 8b 03 <8b> 40 60 8b 00 85 c0 74 17 83 38 02 74 12 8b 80 38 01 00 00 64 
Aug 22 13:22:53 unRAID kernel: EIP: [<c121e878>] scsi_device_get+0x20/0x45 SS:ESP 0068:f1397da8
Aug 22 13:22:53 unRAID kernel: CR2: 00000000737500c9
Aug 22 13:22:53 unRAID kernel: ---[ end trace de264e25dce4930e ]---

Message from syslogd@unRAID at Wed Aug 22 13:22:53 2012 ...
unRAID kernel: Code: 00 00 e8 ff 6e fe ff 5b 5e 5d c3 55 89 e5 53 89 c3 83 b8 54 03 00 00 04 74 2a 8d 80 b8 00 00 00 e8 bc 6e fe ff 85 c0 74 1b 8b 03 <8b> 40 60 8b 00 85 c0 74 17 83 38 02 74 12 8b 80 38 01 00 00 64 

Message from syslogd@unRAID at Wed Aug 22 13:22:53 2012 ...
unRAID kernel: CR2: 00000000737500c9

Message from syslogd@unRAID at Wed Aug 22 13:22:53 2012 ...
unRAID kernel: Oops: 0000 [#1] SMP 

Message from syslogd@unRAID at Wed Aug 22 13:22:53 2012 ...
unRAID kernel: Process smartctl (pid: 6149, ti=f1396000 task=f11f7020 task.ti=f1396000)

Message from syslogd@unRAID at Wed Aug 22 13:22:53 2012 ...
unRAID kernel: Stack:

Message from syslogd@unRAID at Wed Aug 22 13:22:53 2012 ...
unRAID kernel: Call Trace:

Message from syslogd@unRAID at Wed Aug 22 13:22:53 2012 ...
unRAID kernel: EIP: [<c121e878>] scsi_device_get+0x20/0x45 SS:ESP 0068:f1397da8
Aug 22 13:22:53 unRAID logger: ./Media/._TEST.MOV
Aug 22 13:22:53 unRAID logger: *** glibc detected *** rsync: free(): invalid next size (normal): 0x080cbf88 ***
Aug 22 13:22:53 unRAID logger: ======= Backtrace: =========
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(+0x705aa)[0xb769b5aa]
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(+0x73503)[0xb769e503]
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(cfree+0x70)[0xb76a16b0]
Aug 22 13:22:53 unRAID logger: rsync[0x807cd74]
Aug 22 13:22:53 unRAID logger: rsync[0x807de60]
Aug 22 13:22:53 unRAID logger: rsync[0x804f3aa]
Aug 22 13:22:53 unRAID logger: rsync[0x8050bbe]
Aug 22 13:22:53 unRAID logger: rsync[0x8051e56]
Aug 22 13:22:53 unRAID logger: rsync[0x8065825]
Aug 22 13:22:53 unRAID logger: rsync[0x80666ac]
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(__libc_start_main+0xe6)[0xb7641b86]
Aug 22 13:22:53 unRAID logger: rsync[0x804aad1]
Aug 22 13:22:53 unRAID logger: ======= Memory map: ========
Aug 22 13:22:53 unRAID logger: 08048000-0809d000 r-xp 00000000 00:01 2120       /usr/bin/rsync
Aug 22 13:22:53 unRAID logger: 0809d000-080a1000 rw-p 00054000 00:01 2120       /usr/bin/rsync
Aug 22 13:22:53 unRAID logger: 080a1000-080f3000 rw-p 00000000 00:00 0          [heap]
Aug 22 13:22:53 unRAID logger: b7400000-b7421000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b7421000-b7500000 ---p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b75c6000-b75e2000 r-xp 00000000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 22 13:22:53 unRAID logger: b75e2000-b75e3000 rw-p 0001b000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 22 13:22:53 unRAID logger: b75e3000-b7626000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b7626000-b762a000 r-xp 00000000 00:01 545        /lib/libattr.so.1.1.0
Aug 22 13:22:53 unRAID logger: b762a000-b762b000 rw-p 00003000 00:01 545        /lib/libattr.so.1.1.0
Aug 22 13:22:53 unRAID logger: b762b000-b7787000 r-xp 00000000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b7787000-b7788000 ---p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b7788000-b778a000 r--p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b778a000-b778b000 rw-p 0015e000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b778b000-b778e000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b778e000-b7794000 r-xp 00000000 00:01 792        /lib/libpopt.so.0.0.0
Aug 22 13:22:53 unRAID logger: b7794000-b7795000 rw-p 00006000 00:01 792        /lib/libpopt.so.0.0.0
Aug 22 13:22:53 unRAID logger: b7795000-b779b000 r-xp 00000000 00:01 845        /lib/libacl.so.1.1.0
Aug 22 13:22:53 unRAID logger: b779b000-b779c000 rw-p 00005000 00:01 845        /lib/libacl.so.1.1.0
Aug 22 13:22:53 unRAID logger: b77a1000-b77a2000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b77a2000-b77a3000 r-xp 00000000 00:00 0          [vdso]
Aug 22 13:22:53 unRAID logger: b77a3000-b77c0000 r-xp 00000000 00:01 553        /lib/ld-2.11.1.so
Aug 22 13:22:53 unRAID logger: b77c0000-b77c1000 r--p 0001d000 00:01 553        /lib/ld-2.11.1.so
Aug 22 13:22:53 unRAID logger: b77c1000-b77c2000 rw-p 0001e000 00:01 553        /lib/ld-2.11.1.so
Aug 22 13:22:53 unRAID logger: bfdb7000-bfdd8000 rw-p 00000000 00:00 0          [stack]
Aug 22 13:22:53 unRAID logger: find: `rsync' terminated by signal 6
Aug 22 13:22:53 unRAID logger: ./Media/
Aug 22 13:22:53 unRAID logger: rsync: writefd_unbuffered failed to write 79 bytes to socket [Receiver]: Broken pipe (32)
Aug 22 13:22:53 unRAID logger: rsync error: error in rsync protocol data stream (code 12) at io.c(1530) [Receiver=3.0.7]
Aug 22 13:22:53 unRAID logger: *** glibc detected *** rsync: free(): invalid next size (normal): 0x080cbf80 ***
Aug 22 13:22:53 unRAID logger: ======= Backtrace: =========
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(+0x705aa)[0xb76975aa]
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(+0x73503)[0xb769a503]
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(cfree+0x70)[0xb769d6b0]
Aug 22 13:22:53 unRAID logger: rsync[0x807cd74]
Aug 22 13:22:53 unRAID logger: rsync[0x807de60]
Aug 22 13:22:53 unRAID logger: rsync[0x804f3aa]
Aug 22 13:22:53 unRAID logger: rsync[0x8051fa9]
Aug 22 13:22:53 unRAID logger: rsync[0x8065825]
Aug 22 13:22:53 unRAID logger: rsync[0x80666ac]
Aug 22 13:22:53 unRAID logger: /lib/libc.so.6(__libc_start_main+0xe6)[0xb763db86]
Aug 22 13:22:53 unRAID logger: rsync[0x804aad1]
Aug 22 13:22:53 unRAID logger: ======= Memory map: ========
Aug 22 13:22:53 unRAID logger: 08048000-0809d000 r-xp 00000000 00:01 2120       /usr/bin/rsync
Aug 22 13:22:53 unRAID logger: 0809d000-080a1000 rw-p 00054000 00:01 2120       /usr/bin/rsync
Aug 22 13:22:53 unRAID logger: 080a1000-080f3000 rw-p 00000000 00:00 0          [heap]
Aug 22 13:22:53 unRAID logger: b7400000-b7421000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b7421000-b7500000 ---p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b75c2000-b75de000 r-xp 00000000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 22 13:22:53 unRAID logger: b75de000-b75df000 rw-p 0001b000 00:01 2596       /usr/lib/libgcc_s.so.1
Aug 22 13:22:53 unRAID logger: b75df000-b7622000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b7622000-b7626000 r-xp 00000000 00:01 545        /lib/libattr.so.1.1.0
Aug 22 13:22:53 unRAID logger: b7626000-b7627000 rw-p 00003000 00:01 545        /lib/libattr.so.1.1.0
Aug 22 13:22:53 unRAID logger: b7627000-b7783000 r-xp 00000000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b7783000-b7784000 ---p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b7784000-b7786000 r--p 0015c000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b7786000-b7787000 rw-p 0015e000 00:01 842        /lib/libc-2.11.1.so
Aug 22 13:22:53 unRAID logger: b7787000-b778a000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b778a000-b7790000 r-xp 00000000 00:01 792        /lib/libpopt.so.0.0.0
Aug 22 13:22:53 unRAID logger: b7790000-b7791000 rw-p 00006000 00:01 792        /lib/libpopt.so.0.0.0
Aug 22 13:22:53 unRAID logger: b7791000-b7797000 r-xp 00000000 00:01 845        /lib/libacl.so.1.1.0
Aug 22 13:22:53 unRAID logger: b7797000-b7798000 rw-p 00005000 00:01 845        /lib/libacl.so.1.1.0
Aug 22 13:22:53 unRAID logger: b779d000-b779e000 rw-p 00000000 00:00 0 
Aug 22 13:22:53 unRAID logger: b779e000-b779f000 r-xp 00000000 00:00 0          [vdso]
Aug 22 13:22:53 unRAID logger: b779f000-b77bc000 r-xp 00000000 00:01 553        /lib/ld-2.11.1.so
Aug 22 13:22:53 unRAID logger: b77bc000-b77bd000 r--p 0001d000 00:01 553        /lib/ld-2.11.1.so
Aug 22 13:22:53 unRAID logger: b77bd000-b77be000 rw-p 0001e000 00:01 553        /lib/ld-2.11.1.so
Aug 22 13:22:53 unRAID logger: bf863000-bf884000 rw-p 00000000 00:00 0          [stack]
Aug 22 13:22:53 unRAID logger: find: `rsync' terminated by signal 6
Aug 22 13:22:53 unRAID logger: rsync: writefd_unbuffered failed to write 79 bytes to socket [Receiver]: Broken pipe (32)
Aug 22 13:22:53 unRAID logger: rsync error: error in rsync protocol data stream (code 12) at io.c(1530) [Receiver=3.0.7]
Aug 22 13:22:53 unRAID logger: skipping PLEX/
Aug 22 13:22:53 unRAID logger: skipping Usenet/
Aug 22 13:22:53 unRAID logger: mover finished

dgaschk · August 22, 2012

Have you checked the file systems? See my sig.

Rich · August 22, 2012

Yeah, back on 'Reply #22', lol

All disks and cache drive came back 'No corruptions found'

mr-hexen · August 22, 2012

what allocation method, split level, and share details are set?

Rich · August 22, 2012

I have 5 shares...

Files and Programs

Allocation method - H/W

Split level - Blank

Share details - All disks included, using cache drive

Media

Allocation method - H/W

Split level - Blank

Share details - All disks included, currently not using cache drive

PLEX

Allocation method - F/U

Split level - Blank

Share details - All disks excluded, cache only

Usenet

Allocation method - F/U

Split level - Blank

Share details - All disks excluded, cache only

iTunes

Allocation method - F/U

Split level - Blank

Share details - Disk 4 included, disk 1,2,3,5 excluded , using cache

(Mover is working correctly with all shares apart from the 'Media' share; crashing the entire system. This is why 'Media' is currently set to not use the cache drive)

dgaschk · August 22, 2012

What happens if you manually copy files fro the cache to the share?

I have 5 shares...

Files and Programs

Allocation method - H/W

Split level - Blank

Share details - All disks included, using cache drive

Media

Allocation method - H/W

Split level - Blank

Share details - All disks included, currently not using cache drive

PLEX

Allocation method - F/U

Split level - Blank

Share details - All disks excluded, cache only

Usenet

Allocation method - F/U

Split level - Blank

Share details - All disks excluded, cache only

iTunes

Allocation method - F/U

Split level - Blank

Share details - Disk 4 included, disk 1,2,3,5 excluded , using cache

(Mover is working correctly with all shares apart from the 'Media' share; crashing the entire system. This is why 'Media' is currently set to not use the cache drive)

Rich · August 23, 2012

Thats what i have been doing (see Reply #23).

Up until the last few days i thought that mover was solely responsible for crashing the system and didn't know that it was only when it tried to 'move' from the 'Media' share that the problem arose. I scheduled mover to only run once a month to prevent nightly crashes, which started to fill up my cache drive and eventually i had to copy everything from the cache drive to the appropriate disk and folder manually. This didn't cause a problem and the share worked as it should.

The system only crashes when mover is invoked and there are files to move from the cache drive to the 'Media' share... If the 'Media' share is set to not use the cache drive there is not a problem, also if it is set to use the cache drive, but there is nothing to move, there is not a problem $:-\$

All other shares 'move' without a problem.

Frank1940 · August 23, 2012

Thats what i have been doing (see Reply #23).

Up until the last few days i thought that mover was solely responsible for crashing the system and didn't know that it was only when it tried to 'move' from the 'Media' share that the problem arose. I scheduled mover to only run once a month to prevent nightly crashes, which started to fill up my cache drive and eventually i had to copy everything from the cache drive to the appropriate disk and folder manually. This didn't cause a problem and the share worked as it should.

The system only crashes when mover is invoked and there are files to move from the cache drive to the 'Media' share... If the 'Media' share is set to not use the cache drive there is not a problem, also if it is set to use the cache drive, but there is nothing to move, there is not a problem $:-\$

All other shares 'move' without a problem.

When you make the manual move from the cache disk, are you moving to the Share (Media) or are you moving to the disk share (disk1, disk2, etc.)? (I want to be certain and verify exactly what you are doing!)

And what is your 'minimum free space'? (You can find this by clicking on the "Shares" tab, then on the share name (Media in your case).

Rich · August 23, 2012

I'm moving from within the 'Media' folder on the cache drive, to the 'Media' folder on a disk e.g. disk1.

Minimum free space is set to 0

Frank1940 · August 23, 2012

I'm moving from within the 'Media' folder on the cache drive, to the 'Media' folder on a disk e.g. disk1.

Minimum free space is set to 0

As I understand it, you must set the free space on the share (Media in your case) to be LARGER than the Largest file you will move. (If you are moving bluray iso files this would be 50GB!) I understand that that copy will fail if there is not enough room on the drive for the file to fit. Apparently, the underlining operating system looks at the Minimum Free Space setting and see if it is smaller than the actual free space on the drive before it starts the copy. (It does not compare the free space on the drive to the actual file size as there are times when this is unknown-- example, if you are generating an ISO on a another computer with its final destination being in a share in unRAID.)

unRAID will not spilt files across disks when you write to a share. The entire file must fit on the disk that the write starts to. By the way, you may be noticing that I said copy. A 'move' in Linux is a two step operation when it is done between physical disks: first, a copy operation and ,second, a remove operation on the first disk.

I suspect that the script for the Mover may be crashing because it never expects a copy to fail!

You manual copy is succeeding because you are checking to see (or know) that there is sufficient space on the disk that you are copying to!

Rich · August 23, 2012

Ah, i didn't know that! You've just answered that exact question from 'reply #23', thank you.

So am i right in saying that if i set the min free space to 15000 (my biggest files in the 'Media' share being 14.3GB) that should either solve the problem or, at least, eliminate it from the list of potential reasons?

Joe L. · August 23, 2012

Ah, i didn't know that! You've just answered that exact question from 'reply #23', thank you.

So am i right in saying that if i set the min free space to 15000 (my biggest files in the 'Media' share being 14.3GB) that should either solve the problem or, at least, eliminate it from the list of potential reasons?

I think that would set the min to approx 1.5MB. It is in kilobytes. So, 15GB = 15000000 kB.

Rich · August 23, 2012

I think i've managed to fix the problem!!!

After everyones' suggestions and everything i tried, from what i can gather, it was just a corrupted share

I decided to create a new share called 'Entertainment' and go through one disk at a time, cutting the contents of each 'Media' folder to new 'Entertainment' folders. I then deleted the 'Media' share, copied a file to the new 'Entertainment' share, so it was on the cache drive and invoked mover... Worked perfectly!!! I then renamed the 'Entertainment' share to 'Media' and tested again, with no problems

I dont want to get over excited, but it looks like thats the problem completely sorted. I'll keep testing for the next few days as well as setting mover back to running nightly and post back if i get any bad news.

But for now, fingers crossed and thank you all for your time and help,

Rich

mexicanmike · September 2, 2012

OOOh man, I am having this exact same problem and have tried almost everything.

I have eliminated the disk as the issue. Countless replacement disks have the same issue. It is also not the SATA port or SAS card, as I have also moved positions on the board, into different tray locations.

I am running completely stock, and will be attaching system logs on my next post.

Rich,

Did changing min. free space have any impact on this issue. I am about to try changing this to a very large size to see if it helps, but I honestly think I tried this already.

Frank1940 · September 2, 2012

OOOh man, I am having this exact same problem and have tried almost everything.

I have eliminated the disk as the issue. Countless replacement disks have the same issue. It is also not the SATA port or SAS card, as I have also moved positions on the board, into different tray locations.

I am running completely stock, and will be attaching system logs on my next post.

Rich,

Did changing min. free space have any impact on this issue. I am about to try changing this to a very large size to see if it helps, but I honestly think I tried this already.

Read this:

http://lime-technology.com/wiki/index.php/Un-Official_UnRAID_Manual#Min._Free_Space

Mover crashing server

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation