I am receiving an error at the end of cache mover followed by my unread server crashing.

 

Can anyone help decipher the problem or suggest how to fix it?

 

Thanks.

 

Here is the output:

 

Nov 28 04:08:11 unRaid logger: moving "timeMachine"

Nov 28 04:08:11 unRaid logger: ./timeMachine/

Nov 28 04:08:11 unRaid logger: *** Error in `rsync': double free or corruption (!prev): 0x00000000006ca850 ***

Nov 28 04:09:11 unRaid kernel: INFO: rcu_preempt detected stalls on CPUs/tasks: { 2} (detected by 1, t=60002 jiffies, g=10297974, c=10297973, q=8219)

Nov 28 04:09:11 unRaid kernel: Task dump for CPU 2:

Nov 28 04:09:11 unRaid kernel: rsync R running task 0 28859 28858 0x00000008

Nov 28 04:09:11 unRaid kernel: 00000000039386ff 0000000000000286 ffff88000538f920 ffff88000538f8e8

Nov 28 04:09:11 unRaid kernel: ffffffff81083ad9 0000000000000000 0000000000000286 ffff88000538f920

Nov 28 04:09:11 unRaid kernel: 0000000000000000 ffff88000538f908 ffff88000538fa00 ffff88000538fa68

Nov 28 04:09:11 unRaid kernel: Call Trace:

Nov 28 04:09:11 unRaid kernel: [] ? hrtimer_try_to_cancel+0x6b/0x77

Nov 28 04:09:11 unRaid kernel: [] ? search_by_entry_key+0x21/0x22f

Nov 28 04:09:11 unRaid kernel: [] ? reiserfs_find_entry.part.16+0x64/0x2ce

Nov 28 04:09:11 unRaid kernel: [] ? __d_instantiate+0x93/0xf3

Nov 28 04:09:11 unRaid kernel: [] ? d_rehash+0x4f/0x54

Nov 28 04:09:11 unRaid kernel: [] ? d_splice_alias+0x23e/0x254

Nov 28 04:09:11 unRaid kernel: [] ? radix_tree_lookup_slot+0x13/0x22

Nov 28 04:09:11 unRaid kernel: [] ? find_get_entry+0x72/0x7d

Nov 28 04:09:11 unRaid kernel: [] ? pagecache_get_page+0x8a/0x153

Nov 28 04:09:11 unRaid kernel: [] ? bh_lru_install+0xe1/0xee

Nov 28 04:09:11 unRaid kernel: [] ? zone_statistics+0x70/0x77

Nov 28 04:09:11 unRaid kernel: [] ? zone_statistics+0x70/0x77

Nov 28 04:09:11 unRaid kernel: [] ? zone_statistics+0x70/0x77

Nov 28 04:09:11 unRaid kernel: [] ? radix_tree_lookup_slot+0x13/0x22

Nov 28 04:09:11 unRaid kernel: [] ? find_get_entry+0x72/0x7d

Nov 28 04:09:11 unRaid kernel: [] ? find_lock_entry+0x1b/0x5c

Nov 28 04:09:11 unRaid kernel: [] ? shmem_getpage_gfp+0x9f/0x68a

Nov 28 04:09:11 unRaid kernel: [] ? mem_cgroup_begin_page_stat+0x13/0x7a

Nov 28 04:09:11 unRaid kernel: [] ? mem_cgroup_end_page_stat+0x43/0x45

Nov 28 04:09:11 unRaid kernel: [] ? page_add_file_rmap+0x20/0x46

Nov 28 04:09:11 unRaid kernel: [] ? unlock_page+0x1f/0x23

Nov 28 04:09:11 unRaid kernel: [] ? filemap_map_pages+0x5f/0x1d6

Nov 28 04:09:11 unRaid kernel: [] ? handle_mm_fault+0xf82/0x1152

Nov 28 04:09:11 unRaid kernel: [] ? __do_page_fault+0x295/0x309

Nov 28 04:09:11 unRaid kernel: [] ? do_page_fault+0x22/0x2a

Nov 28 04:09:11 unRaid kernel: [] ? page_fault+0x28/0x30

Run a Check Filesystem on all your drives involved.

Thanks for the information.

 

I've checked the filesystem on all drives. Found one drive with an error and followed the instructions to fix it.

 

I still get an error when mover runs, but at least the unraid machine doesn't crash.

 

What next?

 

Thanks again for the help.

 

Now when mover runs I get:

 

Nov 29 11:50:22 unRaid logger: *** Error in `rsync': free(): invalid next size (normal): 0x00000000006ca850 ***

Nov 29 11:50:22 unRaid logger: ======= Backtrace: =========

Nov 29 11:50:22 unRaid logger: /lib64/libc.so.6(+0x7f93a)[0x2acbde42693a]

Nov 29 11:50:22 unRaid logger: rsync[0x43b4fa]

Nov 29 11:50:22 unRaid logger: rsync[0x43ba36]

Nov 29 11:50:22 unRaid logger: rsync[0x408ed9]

Nov 29 11:50:22 unRaid logger: rsync[0x40be71]

Nov 29 11:50:22 unRaid logger: rsync[0x420b8e]

Nov 29 11:50:22 unRaid logger: rsync[0x4049c9]

Nov 29 11:50:22 unRaid logger: /lib64/libc.so.6(__libc_start_main+0xf5)[0x2acbde3c8d05]

Nov 29 11:50:22 unRaid logger: rsync[0x40534d]

Nov 29 11:50:22 unRaid logger: ======= Memory map: ========

Nov 29 11:50:22 unRaid logger: 00400000-0046b000 r-xp 00000000 00:02 3884                              /usr/bin/rsync

Nov 29 11:50:22 unRaid logger: 0066a000-0066f000 rw-p 0006a000 00:02 3884                              /usr/bin/rsync

Nov 29 11:50:22 unRaid logger: 0066f000-006eb000 rw-p 00000000 00:00 0                                  [heap]

Nov 29 11:50:22 unRaid logger: 2acbddd6d000-2acbddd90000 r-xp 00000000 00:02 7525                      /lib64/ld-2.17.so

Nov 29 11:50:22 unRaid logger: 2acbddd90000-2acbddd92000 rw-p 00000000 00:00 0

Nov 29 11:50:22 unRaid logger: 2acbddd9b000-2acbdde21000 rw-p 00000000 00:00 0

Nov 29 11:50:22 unRaid logger: 2acbddf90000-2acbddf91000 r--p 00023000 00:02 7525                      /lib64/ld-2.17.so

Nov 29 11:50:22 unRaid logger: 2acbddf91000-2acbddf93000 rw-p 00024000 00:02 7525                      /lib64/ld-2.17.so

Nov 29 11:50:22 unRaid logger: 2acbddf93000-2acbddf9a000 r-xp 00000000 00:02 7548                      /lib64/libacl.so.1.1.0

Nov 29 11:50:22 unRaid logger: 2acbddf9a000-2acbde19a000 ---p 00007000 00:02 7548                      /lib64/libacl.so.1.1.0

Nov 29 11:50:22 unRaid logger: 2acbde19a000-2acbde19b000 rw-p 00007000 00:02 7548                      /lib64/libacl.so.1.1.0

Nov 29 11:50:22 unRaid logger: 2acbde19b000-2acbde1a6000 r-xp 00000000 00:02 7488                      /lib64/libpopt.so.0.0.0

Nov 29 11:50:22 unRaid logger: 2acbde1a6000-2acbde3a6000 ---p 0000b000 00:02 7488                      /lib64/libpopt.so.0.0.0

Nov 29 11:50:22 unRaid logger: 2acbde3a6000-2acbde3a7000 rw-p 0000b000 00:02 7488                      /lib64/libpopt.so.0.0.0

Nov 29 11:50:22 unRaid logger: 2acbde3a7000-2acbde566000 r-xp 00000000 00:02 7448                      /lib64/libc-2.17.so

Nov 29 11:50:22 unRaid logger: 2acbde566000-2acbde766000 ---p 001bf000 00:02 7448                      /lib64/libc-2.17.so

Nov 29 11:50:22 unRaid logger: 2acbde766000-2acbde76a000 r--p 001bf000 00:02 7448                      /lib64/libc-2.17.so

Nov 29 11:50:22 unRaid logger: 2acbde76a000-2acbde76c000 rw-p 001c3000 00:02 7448                      /lib64/libc-2.17.so

Nov 29 11:50:22 unRaid logger: 2acbde76c000-2acbde771000 rw-p 00000000 00:00 0

Nov 29 11:50:22 unRaid logger: 2acbde771000-2acbde775000 r-xp 00000000 00:02 7468                      /lib64/libattr.so.1.1.0

Nov 29 11:50:22 unRaid logger: 2acbde775000-2acbde974000 ---p 00004000 00:02 7468                      /lib64/libattr.so.1.1.0

Nov 29 11:50:22 unRaid logger: 2acbde974000-2acbde975000 rw-p 00003000 00:02 7468                      /lib64/libattr.so.1.1.0

Nov 29 11:50:22 unRaid logger: 2acbde975000-2acbde98a000 r-xp 00000000 00:02 5098                      /usr/lib64/libgcc_s.so.1

Nov 29 11:50:22 unRaid logger: 2acbde98a000-2acbdeb8a000 ---p 00015000 00:02 5098                      /usr/lib64/libgcc_s.so.1

Nov 29 11:50:22 unRaid logger: 2acbdeb8a000-2acbdeb8b000 rw-p 00015000 00:02 5098                      /usr/lib64/libgcc_s.so.1

Nov 29 11:50:22 unRaid logger: 7ffdb9b7d000-7ffdb9b9e000 rw-p 00000000 00:00 0                          [stack]

Nov 29 11:50:22 unRaid logger: 7ffdb9bd2000-7ffdb9bd4000 r--p 00000000 00:00 0                          [vvar]

Nov 29 11:50:22 unRaid logger: 7ffdb9bd4000-7ffdb9bd6000 r-xp 00000000 00:00 0                          [vdso]

Nov 29 11:50:22 unRaid logger: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Nov 29 11:50:22 unRaid logger: find: `rsync' terminated by signal 6

Nov 29 11:50:22 unRaid logger: mover finished