March 9, 200917 yr I use my unRAID server for backup up all my data to it, but the SMB shares keep stoping. They just disappear. So then i wrote in the crontab to reboot daily, that seemed to fix the problem, but it seems if i copy a few hundred gigs to it the shares stop again. What should I do and what logs to you guys need to see? Thanks, Coz
March 10, 200917 yr grab a syslog that covers a period where it stops. upload that syslog to pastebin if long or here if short. then wait for a reply
March 11, 200917 yr Author Mar 10 19:46:06 BlackHole kernel: smbd invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0 Mar 10 19:46:06 BlackHole kernel: Pid: 1388, comm: smbd Not tainted 2.6.27.7-unRAID #3 Mar 10 19:46:06 BlackHole kernel: [<c013dd20>] oom_kill_process+0x4e/0x19b Mar 10 19:46:06 BlackHole kernel: [<c013e144>] out_of_memory+0x14e/0x17f Mar 10 19:46:06 BlackHole kernel: [<c013fee8>] __alloc_pages_internal+0x2cd/0x372 Mar 10 19:46:06 BlackHole kernel: [<c013ff9e>] __get_free_pages+0x11/0x1e Mar 10 19:46:06 BlackHole kernel: [<c011b2e7>] copy_process+0x89/0xd8b Mar 10 19:46:06 BlackHole kernel: [<c011a251>] task_new_fair+0x7b/0x8e Mar 10 19:46:06 BlackHole kernel: [<c011c121>] do_fork+0xcc/0x215 Mar 10 19:46:06 BlackHole kernel: [<c01015f5>] sys_clone+0x22/0x26 Mar 10 19:46:06 BlackHole kernel: [<c0102c3e>] syscall_call+0x7/0xb Mar 10 19:46:06 BlackHole kernel: [<c0310000>] quirk_piix4_acpi+0x5e/0x13c Mar 10 19:46:06 BlackHole kernel: ======================= Mar 10 19:46:06 BlackHole kernel: Mem-Info: Mar 10 19:46:06 BlackHole kernel: DMA per-cpu: Mar 10 19:46:06 BlackHole kernel: CPU 0: hi: 0, btch: 1 usd: 0 Mar 10 19:46:06 BlackHole kernel: CPU 1: hi: 0, btch: 1 usd: 0 Mar 10 19:46:06 BlackHole kernel: Normal per-cpu: Mar 10 19:46:06 BlackHole kernel: CPU 0: hi: 186, btch: 31 usd: 0 Mar 10 19:46:06 BlackHole kernel: CPU 1: hi: 186, btch: 31 usd: 0 Mar 10 19:46:06 BlackHole kernel: HighMem per-cpu: Mar 10 19:46:06 BlackHole kernel: CPU 0: hi: 186, btch: 31 usd: 0 Mar 10 19:46:06 BlackHole kernel: CPU 1: hi: 186, btch: 31 usd: 0 Mar 10 19:46:06 BlackHole kernel: Active:27807 inactive:119221 dirty:0 writeback:0 unstable:0 Mar 10 19:46:06 BlackHole kernel: free:192427 slab:173164 mapped:732 pagetables:149 bounce:0 Mar 10 19:46:06 BlackHole kernel: DMA free:3548kB min:64kB low:80kB high:96kB active:88kB inactive:40kB present:15796kB pages_scanned:0 all_unreclaimable? no Mar 10 19:46:06 BlackHole kernel: lowmem_reserve[]: 0 873 2015 2015 Mar 10 19:46:06 BlackHole kernel: Normal free:91088kB min:3744kB low:4680kB high:5616kB active:42252kB inactive:48136kB present:894080kB pages_scanned:0 all_unreclaimable? no Mar 10 19:46:06 BlackHole kernel: lowmem_reserve[]: 0 0 9140 9140 Mar 10 19:46:06 BlackHole kernel: HighMem free:675072kB min:512kB low:1736kB high:2960kB active:68888kB inactive:428708kB present:1169924kB pages_scanned:0 all_unreclaimable? no Mar 10 19:46:06 BlackHole kernel: lowmem_reserve[]: 0 0 0 0 Mar 10 19:46:06 BlackHole kernel: DMA: 23*4kB 14*8kB 25*16kB 26*32kB 25*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3548kB Mar 10 19:46:06 BlackHole kernel: Normal: 22280*4kB 39*8kB 1*16kB 2*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 91176kB Mar 10 19:46:06 BlackHole kernel: HighMem: 20*4kB 27*8kB 28*16kB 31*32kB 2068*64kB 2089*128kB 803*256kB 129*512kB 2*1024kB 0*2048kB 0*4096kB = 675144kB Mar 10 19:46:06 BlackHole kernel: 140564 total pagecache pages Mar 10 19:46:06 BlackHole kernel: 0 pages in swap cache Mar 10 19:46:06 BlackHole kernel: Swap cache stats: add 0, delete 0, find 0/0 Mar 10 19:46:06 BlackHole kernel: Free swap = 0kB Mar 10 19:46:06 BlackHole kernel: Total swap = 0kB Mar 10 19:46:06 BlackHole kernel: 524144 pages RAM Mar 10 19:46:06 BlackHole kernel: 294784 pages HighMem Mar 10 19:46:06 BlackHole kernel: 5327 pages reserved Mar 10 19:46:06 BlackHole kernel: 68675 pages shared Mar 10 19:46:06 BlackHole kernel: 260048 pages non-shared Mar 10 19:46:06 BlackHole kernel: Out of memory: kill process 1466 (shfs) score 557 or a child Mar 10 19:46:06 BlackHole kernel: Killed process 1466 (shfs)
March 11, 200917 yr How much memory do you have and are you running anything other than the stock raid install
March 11, 200917 yr its this line which jumps out Out of memory: kill process 1466 (shfs) score 557 or a child dont have time to look properly this now but the syslog doesnt seem to match up with you saying you have 2GB of ram. do some poking about and see
March 11, 200917 yr its this line which jumps out Out of memory: kill process 1466 (shfs) score 557 or a child dont have time to look properly this now but the syslog doesnt seem to match up with you saying you have 2GB of ram. do some poking about and see It is obvious you ran out of memory and the user-share file system process was killed by the OS trying to free up some space. Because you only showed the tail end of the syslog, we do not have a clue why the memory was used up. Usually, it is because the syslog fills and uses up all free space, but any process can allocate memory, forget to free it, and eventually use up all memory.. Can you post the portion of the syslog before the section you posted? (Odds are it filled to where it used up all your ram) What specific version of unRAID are you running? Are you running ANY add-ons, or scripts you have added? Joe L.
March 11, 200917 yr Author the syslog is 13.5 megs here is more before when it happened. and i'm running 4.4.2 ok, i have unmenu installed Syslog of when it happened http://prime-trading.com/syslog.txt My Go script http://prime-trading.com/go.txt My crontab -l http://prime-trading.com/crontab.txt And the script for my nightly reboot which is a small modification to the powerdown script copied from here http://prime-trading.com/softreboot.txt
March 11, 200917 yr Your mover logged a lot, but honestly I don't think 13.5M would have a negative effect on memory since you have 2G. The mover ended some 12 hours before the log activity indicating that the system ran out of memory (see log snippet below). I'm interested in learning what happened here ... Mar 10 06:57:59 BlackHole logger: mover finished Mar 10 19:46:06 BlackHole kernel: smbd invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0 Mar 10 19:46:06 BlackHole kernel: Pid: 1388, comm: smbd Not tainted 2.6.27.7-unRAID #3 Mar 10 19:46:06 BlackHole kernel: [<c013dd20>] oom_kill_process+0x4e/0x19b Mar 10 19:46:06 BlackHole kernel: [<c013e144>] out_of_memory+0x14e/0x17f Mar 10 19:46:06 BlackHole kernel: [<c013fee8>] __alloc_pages_internal+0x2cd/0x372 Mar 10 19:46:06 BlackHole kernel: [<c013ff9e>] __get_free_pages+0x11/0x1e Mar 10 19:46:06 BlackHole kernel: [<c011b2e7>] copy_process+0x89/0xd8b Mar 10 19:46:06 BlackHole kernel: [<c011a251>] task_new_fair+0x7b/0x8e Mar 10 19:46:06 BlackHole kernel: [<c011c121>] do_fork+0xcc/0x215 Mar 10 19:46:06 BlackHole kernel: [<c01015f5>] sys_clone+0x22/0x26 Mar 10 19:46:06 BlackHole kernel: [<c0102c3e>] syscall_call+0x7/0xb Mar 10 19:46:06 BlackHole kernel: [<c0310000>] quirk_piix4_acpi+0x5e/0x13c Mar 10 19:46:06 BlackHole kernel: ======================= Mar 10 19:46:06 BlackHole kernel: Mem-Info: Mar 10 19:46:06 BlackHole kernel: DMA per-cpu: Mar 10 19:46:06 BlackHole kernel: CPU 0: hi: 0, btch: 1 usd: 0 Mar 10 19:46:06 BlackHole kernel: CPU 1: hi: 0, btch: 1 usd: 0 Mar 10 19:46:06 BlackHole kernel: Normal per-cpu: Mar 10 19:46:06 BlackHole kernel: CPU 0: hi: 186, btch: 31 usd: 0 Mar 10 19:46:06 BlackHole kernel: CPU 1: hi: 186, btch: 31 usd: 0 Mar 10 19:46:06 BlackHole kernel: HighMem per-cpu: Mar 10 19:46:06 BlackHole kernel: CPU 0: hi: 186, btch: 31 usd: 0 Mar 10 19:46:06 BlackHole kernel: CPU 1: hi: 186, btch: 31 usd: 0 Mar 10 19:46:06 BlackHole kernel: Active:27807 inactive:119221 dirty:0 writeback:0 unstable:0 Mar 10 19:46:06 BlackHole kernel: free:192427 slab:173164 mapped:732 pagetables:149 bounce:0 Mar 10 19:46:06 BlackHole kernel: DMA free:3548kB min:64kB low:80kB high:96kB active:88kB inactive:40kB present:15796kB pages_scanned:0 all_unreclaimable? no Mar 10 19:46:06 BlackHole kernel: lowmem_reserve[]: 0 873 2015 2015 Mar 10 19:46:06 BlackHole kernel: Normal free:91088kB min:3744kB low:4680kB high:5616kB active:42252kB inactive:48136kB present:894080kB pages_scanned:0 all_unreclaimable? no Mar 10 19:46:06 BlackHole kernel: lowmem_reserve[]: 0 0 9140 9140 Mar 10 19:46:06 BlackHole kernel: HighMem free:675072kB min:512kB low:1736kB high:2960kB active:68888kB inactive:428708kB present:1169924kB pages_scanned:0 all_unreclaimable? no Mar 10 19:46:06 BlackHole kernel: lowmem_reserve[]: 0 0 0 0 Mar 10 19:46:06 BlackHole kernel: DMA: 23*4kB 14*8kB 25*16kB 26*32kB 25*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3548kB Mar 10 19:46:06 BlackHole kernel: Normal: 22280*4kB 39*8kB 1*16kB 2*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 91176kB Mar 10 19:46:06 BlackHole kernel: HighMem: 20*4kB 27*8kB 28*16kB 31*32kB 2068*64kB 2089*128kB 803*256kB 129*512kB 2*1024kB 0*2048kB 0*4096kB = 675144kB Mar 10 19:46:06 BlackHole kernel: 140564 total pagecache pages Mar 10 19:46:06 BlackHole kernel: 0 pages in swap cache Mar 10 19:46:06 BlackHole kernel: Swap cache stats: add 0, delete 0, find 0/0 Mar 10 19:46:06 BlackHole kernel: Free swap = 0kB Mar 10 19:46:06 BlackHole kernel: Total swap = 0kB Mar 10 19:46:06 BlackHole kernel: 524144 pages RAM Mar 10 19:46:06 BlackHole kernel: 294784 pages HighMem Mar 10 19:46:06 BlackHole kernel: 5327 pages reserved Mar 10 19:46:06 BlackHole kernel: 68675 pages shared Mar 10 19:46:06 BlackHole kernel: 260048 pages non-shared Mar 10 19:46:06 BlackHole kernel: Out of memory: kill process 1466 (shfs) score 557 or a child Mar 10 19:46:06 BlackHole kernel: Killed process 1466 (shfs) Mar 10 20:44:53 BlackHole in.telnetd[26347]: connect from 10.10.0.182 (10.10.0.182)
March 11, 200917 yr Author I use the server for backing up data from all around the network, 1 computer on the network pull data from 1 computer at a time, and puts it on the unRAID system via a network drive share. Nightly that about 10 gigs of data copied to it, and on the weekends its 300gigs. It has 8 hard drives, they are all SATA 750gigs, 1 is cache, 1 is parity, leaving 6 for parity. Also to note, my backup scripts where completed 2009-03-10 Tue 19:31:40 and it was a small copy 10megs script. If you need any more info let me know. and Joe or NAS if you need remote access to the system i should be able to let you. Specs: Motheboard: ASUS P5QL Pro Chip: Intel Duo E2200 2.2ghz Ram: 2gigs
March 11, 200917 yr I use the server for backing up data from all around the network, 1 computer on the network pull data from 1 computer at a time, and puts it on the unRAID system via a network drive share. Nightly that about 10 gigs of data copied to it, and on the weekends its 300gigs. It has 8 hard drives, they are all SATA 750gigs, 1 is cache, 1 is parity, leaving 6 for parity data. Also to note, my backup scripts where completed 2009-03-10 Tue 19:31:40 and it was a small copy 10megs script. If you need any more info let me know. and Joe or NAS if you need remote access to the system i should be able to let you. What is the "winexe" program you are copying into place in the "go" script? Is it running? Are your backups to a user-share? or directly to a disk share? Are your backup files all in a sub-directory? or all at the "root" level of a disk or user share? Are you writing to a share that has the "cache" drive enabled? (In other words, is it involved at all when the crash occurred?) There are only a few processes involved. The fact that you must reboot nightly to free up memory is very unusual. It indicates to me that some process on your server is allocating memory and not freeing it, or your system log is growing to where it uses up all the free memory. Since the syslog is not too large, I'm guessing it must be either the samba daemon itself or shfs itself. The only way to know for sure is to see which process is growing in its memory usage over time. Typing this command after logging in via telnet will list the processes on the server and their memory size (first column) sorted by the memory size. ps -eo size,pid,args --sort size In general, processes should not grow much once they allocate their initial memory. If you run the "ps" command above periodically while your backup is occurring you might be able to spot the process that is using up all your RAM. (Its memory size will continuously increase) On my server, the biggest user of memory is the shfs process. Typing: free will give you an idea of the memory usage at any given time. It is possible that a log file is using up all the free memory instead of a process. That would show as a gradual drop in the free and buffered memory and no growth in process size. Joe L.
March 11, 200917 yr Author Winexe is a remote executing program, its not in use, nor is it running. It was something i was looking into for some a script i was planning to create. So it could be deleted, cause it has no impact. The backups are going to a user share. Sub folders ie: \\Tower\Backups\2009-03-11\ And cache drive is enabled for user share "Backups" As far as the ps and free, should i create a bash file that does the ps and free with time stamp and add that to a cron tab?
March 12, 200917 yr There might be a memory leak somewhere in shfs. There seems to be allot of files being written to cache, then moved Maybe you could try writing your backup to a disk share directly (and not rebooting) to see if you continue to be affected.
March 14, 200917 yr Author I've disabled the cache on most of my user shares that do allot of traffic. And this has seem to prevent the problem from showing up.
April 3, 200917 yr Author Ok, so it was still crashing, but I'm sure I finally figured it out. I use the ls-R script that cache's my directory structure into they ram. Well my directory structure is huge, so I'm thinking that was filling up my ram, and then causing it to crash.
Archived
This topic is now archived and is closed to further replies.