February 17, 201313 yr Last week I upgraded from 4.4.2 to 4.7 to 5.0rc11 and all has been fine until yesterday. I also recently installed the MySQL plugin to sync my XBMC library, which works great. I have 11 disks (9 data, 1 parity and 1 cache). Been moving a good amount data around and I am going to be upgrading several drives so I wanted to backup my data first, before I start upgrading drives and moving some disks between different controllers. I have twice in the last 24 hours had my system crash (kernel errors in log, won't respond to telnet and won't respond to powerdown command on console). The last time was this morning, here's what I did: 1. Normal Powerdown. 2. Connect USB drive and add new disk to server. 3. Start system. 4. Mount external USB drive with ntfs. 5. Start rsync to copy data to external USB drive. 6. Start preclear on new disk. 7. System crashed almost immediately. 8. Powerdown did nothing. Interestingly, if I went to /usr/sbin and ran "sh powerdown" and then it ran through some of the process, but did not turn off. 9. "shutdown -now" and once got to single user mode I power cycled the system. Syslog attached. I was gone most of the day, so I been running memtest for the last 11 hours and looks fine. Any ideas on why my system is crashing? I still want to backup my data, so I guess I am going connect a disk, but not add to the array, and format so that I can backup data directly to the drive without using ntfs. Next resort would be to backup over the network, but that would take a really long time. FYI - My server build is here: http://lime-technology.com/forum/index.php?topic=3817.msg33483#msg33483 syslog.txt
February 17, 201313 yr You have lack of memory, the OOM (Out Of Memory) killer has killed of mysqld (mysql) to get memory back and support the more active processes. The kernel in V5 handles memory differently and OOM errors happen more, the fact that you also installed mysqld also gives a higher effect on memory use. How much memory are you running with ? It sounds weird but having more then 4gig of memory actually makes it MORE likely to get memory errors like this. You are experiencing lack in LOWER memory, so adding memory does not help. Even worse: The more memory you put in your system the more LOWER memory is beiing used to be able to map it. Search the forum for the MEM=4095 parameter and try to boot with that enabled, it might help (it partly "fools" the system in thinking you have only 4gb on board). Going back to 4gb of physical RAM has solved the OOM errors for me. Running with 16gig I had processes dying on a weekly basis. With 4gig of ram this has not happened since the change (And that was a few weeks ago). I am still running the same plugins and am running the system in the same way.
February 17, 201313 yr Author I only have 4GB of memory installed. How can I tell what is using all the lower memory?
February 17, 201313 yr Author I am currently preclearing 2 disks and copying a bunch of data off the array, so lots of reads. Here's what it looks like right now. Any ideas? root@voyager:~# free -lm total used free shared buffers cached Mem: 4016 3980 35 0 280 3038 Low: 846 837 8 High: 3169 3142 26 -/+ buffers/cache: 661 3354 Swap: 0 0 0 root@voyager:~# egrep 'High|Low' /proc/meminfo HighTotal: 3245640 kB HighFree: 30140 kB LowTotal: 867096 kB LowFree: 8140 kB root@voyager:~# vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 3 0 15868 288316 3130748 0 0 2446 57 1830 1963 3 22 6 69 0 2 0 15816 288260 3131980 0 0 121589 0 3637 5274 6 18 10 67 0 2 0 29396 287388 3119536 0 0 120675 0 3428 4598 4 18 5 72 0 5 0 28104 272964 3130400 0 0 120279 5 3918 5853 7 20 5 68 0 2 0 16092 287492 3132020 0 0 121817 5 4057 6344 4 21 9 66 root@voyager:~# top top - 00:45:16 up 2:24, 3 users, load average: 3.46, 3.54, 3.49 Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie Cpu(s): 7.8%us, 15.7%sy, 0.0%ni, 8.6%id, 60.8%wa, 0.0%hi, 7.1%si, 0.0%st Mem: 4112736k total, 4096040k used, 16696k free, 285880k buffers Swap: 0k total, 0k used, 0k free, 3132708k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3878 root 20 0 108m 48m 576 S 10 1.2 10:28.56 shfs 4024 mysql 20 0 122m 18m 5320 S 0 0.5 0:06.92 mysqld 4446 jimmyjoe 20 0 22612 9.8m 8272 S 19 0.2 14:24.30 smbd 9840 root 20 0 10348 8608 612 D 9 0.2 0:02.06 dd 10022 root 20 0 10348 8608 612 D 9 0.2 0:01.55 dd 4047 root 20 0 15712 3708 2980 S 0 0.1 0:00.09 smbd 17397 root 20 0 4680 3436 936 S 0 0.1 0:00.24 awk 4045 root 20 0 9192 1924 1376 S 0 0.0 0:00.10 nmbd 4506 root 20 0 6844 1920 1484 S 0 0.0 0:00.02 in.telnetd 6202 root 20 0 4596 1908 1168 S 0 0.0 0:01.32 preclear_disk.s 7559 root 20 0 4596 1908 1168 S 0 0.0 0:01.62 preclear_disk.s 4507 root 20 0 4304 1740 1320 S 0 0.0 0:00.03 bash 4152 root 20 0 4320 1732 1296 S 0 0.0 0:00.02 bash 4153 root 20 0 4320 1732 1296 S 0 0.0 0:00.01 bash 4051 root 20 0 15712 1596 868 S 0 0.0 0:00.01 smbd 1301 root 20 0 4492 1540 1216 S 0 0.0 0:00.43 ntpd 1662 root 20 0 11836 1496 1180 S 0 0.0 0:01.22 emhttp 1969 root 20 0 2808 1440 1100 S 0 0.0 0:02.11 unraid_notify 3912 root 20 0 2668 1268 1068 S 0 0.0 0:00.01 mysqld_safe 3867 root 20 0 28724 1256 556 S 0 0.0 0:02.38 shfs 9450 root 20 0 2464 1076 824 R 0 0.0 0:00.08 top 706 root 16 -4 2472 1072 528 S 0 0.0 0:00.20 udevd 9839 root 20 0 4596 1064 320 S 0 0.0 0:00.00 preclear_disk.s 10021 root 20 0 4596 1064 320 S 0 0.0 0:00.00 preclear_disk.s 4162 root 18 -2 2468 1052 496 S 0 0.0 0:01.09 udevd 9832 root 20 0 4596 1040 296 S 0 0.0 0:00.00 preclear_disk.s 10015 root 20 0 4596 1040 296 S 0 0.0 0:00.00 preclear_disk.s 4145 root 20 0 2920 1032 580 S 0 0.0 0:02.14 cache_dirs 4212 root 18 -2 2468 1032 484 S 0 0.0 0:00.00 udevd 9842 root 20 0 2588 912 704 S 0 0.0 0:00.00 awk 10024 root 20 0 2588 912 704 S 0 0.0 0:00.00 awk 1284 root 20 0 2108 788 660 S 0 0.0 0:00.01 rpc.statd 2291 root 20 0 10276 648 512 S 0 0.0 0:02.24 apcupsd 1308 root 20 0 1864 632 536 S 0 0.0 0:00.00 acpid 9841 root 20 0 2244 624 464 S 0 0.0 0:00.00 sed 10023 root 20 0 2244 624 464 S 0 0.0 0:00.00 sed 1171 root 20 0 1908 604 524 S 0 0.0 0:00.04 syslogd 1323 root 20 0 1908 584 504 S 0 0.0 0:00.01 crond 4062 root 20 0 2644 540 368 S 0 0.0 0:00.00 uu 1294 root 20 0 1900 536 468 S 0 0.0 0:00.00 inetd 4155 root 20 0 1856 536 472 S 0 0.0 0:00.00 agetty 4157 root 20 0 1856 536 472 S 0 0.0 0:00.00 agetty 4154 root 20 0 1856 532 472 S 0 0.0 0:00.00 agetty 4156 root 20 0 1856 528 472 S 0 0.0 0:00.00 agetty 4063 root 20 0 1840 520 456 S 0 0.0 0:00.00 logger 1280 bin 20 0 1960 476 400 S 0 0.0 0:00.00 rpc.portmap 1175 root 20 0 1856 384 320 S 0 0.0 0:00.00 klogd 1318 messageb 20 0 2476 364 212 S 0 0.0 0:00.00 dbus-daemon 1 root 20 0 828 308 264 S 0 0.0 0:05.64 init 1325 daemon 20 0 1900 288 220 S 0 0.0 0:00.00 atd 10140 root 20 0 1852 244 204 S 0 0.0 0:00.00 sleep 10049 root 20 0 1852 240 204 S 0 0.0 0:00.00 sleep 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0 0.0 0:01.56 ksoftirqd/0 6 root RT 0 0 0 0 S 0 0.0 0:00.07 migration/0
February 17, 201313 yr MMm... I am not knowledgeable enough to answer that question.... I would advise making a few OOM adjust settings that will at least keep the unraid webpages active... Search for OOM_Adjust and OOM_Score on the forum.. Preclearing has stressed my memory also so you might try preclearing one at a time..
February 17, 201313 yr Preclearing has stressed my memory also so you might try preclearing one at a time.. Or, use the -w, -r and -b options to limit the memory used. preclear_disk.sh -w 65536 -r 65536 -b 200 /dev/sdX -w size = write block size in bytes -r size = read block size in bytes -b count = number of blocks to read at a time
February 18, 201313 yr Author Well, the first preclear read finished in about 24 hours and then went to start writing, and I got the following errors and it was writing REALLY slow after this: Feb 17 17:22:22 voyager udevd[706]: worker [4212] unexpectedly returned with status 0x0100 Feb 17 17:22:22 voyager udevd[706]: worker [4212] failed while handling '/devices/pci0000:00/0000:00:1e.0/0000:05:01.0/0000:06:00.0/ata11/host9/target9:0:0/9:0:0:0/block/sdi' (Minor Issues) Feb 17 17:24:16 voyager udevd[706]: worker [4162] unexpectedly returned with status 0x0100 Feb 17 17:24:16 voyager udevd[706]: worker [4162] failed while handling '/devices/pci0000:00/0000:00:1e.0/0000:05:01.0/0000:06:00.0/ata12/host10/target10:0:0/10:0:0:0/block/sdj' (Minor Issues) So I killed them both. I'll either try just running one at a time, or the settings Joe recommended. Thanks
Archived
This topic is now archived and is closed to further replies.