April 3, 201115 yr So I finally got the unRaid server setup a couple of days ago. Copied all my data over. Now I'm starting to use it as I did with my old DLINK 343 NAS which has served me pretty well, albeit a bit slow. Here's the problem: Last night, I'm streaming a TV show over network to my WDTV player. The show stops all of a sudden and I find out that the unRaid server crashed... hard. No console video output, would not even respond to pings... so I had to hard reboot. After rebooting, checked syslog, they looked fine. I do notice that free memory was slowly being used... mostly to the cache, which is not uncommon with linux. I finish watching rest of my TV show while monitoring the free memory and while it was slowly decreasing it hit 300Mb when I finished the show. This morning I check on the unRaid server and here is what I find: root@Tower:/boot/scripts# free total used free shared buffers cached Mem: 4115664 4097292 18372 0 731088 3221064 -/+ buffers/cache: 145140 3970524 Swap: 0 0 0 top - 06:55:46 up 8:10, 5 users, load average: 0.52, 0.43, 0.33 Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie Cpu(s): 0.4%us, 10.2%sy, 0.0%ni, 42.3%id, 45.5%wa, 0.2%hi, 1.5%si, 0.0%st Mem: 4115664k total, 4097820k used, 17844k free, 730824k buffers Swap: 0k total, 0k used, 0k free, 3221116k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1030 root 20 0 0 0 0 S 9 0.0 7:42.63 flush-8:48 15617 root 20 0 0 0 0 S 8 0.0 5:24.99 flush-8:112 21082 root 20 0 0 0 0 S 7 0.0 8:34.90 flush-8:96 332 root 20 0 0 0 0 S 3 0.0 11:58.92 kswapd0 15785 root 20 0 3844 2708 568 S 2 0.1 1:03.34 dd 1191 root 20 0 3840 2704 568 S 1 0.1 1:12.32 dd 1623 root 20 0 4944 3900 892 S 1 0.1 0:19.40 awk 21215 root 20 0 3840 2704 568 S 1 0.1 1:30.20 dd 30241 root 20 0 2116 1032 788 R 1 0.0 0:00.02 top 1 root 20 0 700 304 264 S 0 0.0 0:01.85 init 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root RT 0 0 0 0 S 0 0.0 0:00.29 migration/0 4 root 20 0 0 0 0 S 0 0.0 0:00.02 ksoftirqd/0 5 root RT 0 0 0 0 S 0 0.0 0:00.44 migration/1 6 root 20 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1 7 root RT 0 0 0 0 S 0 0.0 0:03.58 migration/2 8 root 20 0 0 0 0 S 0 0.0 0:00.01 ksoftirqd/2 9 root 20 0 0 0 0 S 0 0.0 0:00.00 events/0 10 root 20 0 0 0 0 S 0 0.0 0:00.00 events/1 11 root 20 0 0 0 0 S 0 0.0 0:00.00 events/2 12 root 20 0 0 0 0 S 0 0.0 0:00.01 khelper 17 root 20 0 0 0 0 S 0 0.0 0:00.00 async/mgr 120 root 20 0 0 0 0 S 0 0.0 0:05.30 sync_supers 122 root 20 0 0 0 0 S 0 0.0 0:00.56 bdi-default 124 root 20 0 0 0 0 S 0 0.0 0:00.00 kblockd/0 125 root 20 0 0 0 0 S 0 0.0 0:00.03 kblockd/1 126 root 20 0 0 0 0 S 0 0.0 0:00.24 kblockd/2 127 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpid 128 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpi_notify 129 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpi_hotplug 241 root 20 0 0 0 0 S 0 0.0 0:00.00 ata/0 242 root 20 0 0 0 0 S 0 0.0 0:00.00 ata/1 243 root 20 0 0 0 0 S 0 0.0 0:00.00 ata/2 244 root 20 0 0 0 0 S 0 0.0 0:00.00 ata_aux 248 root 20 0 0 0 0 S 0 0.0 0:00.00 ksuspend_usbd 253 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd 256 root 20 0 0 0 0 S 0 0.0 0:00.00 kseriod 291 root 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/0 292 root 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/1 293 root 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/2 My uTorrent client on my desktop PC which directly saves to unRaid is running but it claims the disk is overloaded and half of the torrents show big red X and that the directory is unavailable, seems like unRaid timed out on a few torrents. When I go to restart all of the torrents (10) it works fine for a bit but then speed drops to <30kb/s down from 600kb/s and again, the disk overloaded message appears on the bottom of uTorrent. On unRaid I killed cache_dirs wondering if that was causing issue, but no effect. I do have 3 preclears running on unRaid at the moment. (it was preclearing when I was streaming the TV show, had it going with torrents too) Could this be causing some of this ? Is it the cache that is using up all of the memory ? showing 18M free memory seems really really low. The syslog is attached and is fine, not big at all. I even have my log file on a tmp ram drive. root@Tower:/boot/scripts# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 3907568 131212 3776356 4% /boot tmpfs 2057832 84 2057748 1% /tmp/log /dev/md4 488371640 246929796 241441844 51% /mnt/disk4 /dev/md2 1953454928 1831607096 121847832 94% /mnt/disk2 /dev/md3 488371640 341668268 146703372 70% /mnt/disk3 /dev/md1 1953454928 1831331836 122123092 94% /mnt/disk1 shfs 4883653136 4251536996 632116140 88% /mnt/user as you can see only very small part of memory is used to hold syslog. My problem right now is why uTorrent is showing disk overloaded, seems like the IO is very slow from unRaid so it has to throttle my torrents back to accomodate. I'm stopped uTorrent and even accessing files on the server is slow... I'm trying to copy a directory to my desktop PC and it's transferring at 173kb/s!!! syslog-2011-04-03.zip
April 3, 201115 yr How big does the log file get? It is on a RAM file so if it gets too big it will fill RAM and crash the server.
April 3, 201115 yr Author i'm not sure since i couldn't recover the syslog due to the crash... but right now my log file is tiny, around 75k... I've got 3 preclears going but the fastest I can read from the unRaid server is about 200kb/s... which is far shy of the 55Mb/s I was getting the other day...
April 4, 201115 yr Pre-clears will effect array performance because all drives share the system buss. This does seem like extremely poor performance though. Post you system specs and the entire syslog. Zip it if needed.
April 4, 201115 yr Author Weird... one of my preclears just finished and it failed, gave me the following: ================================================================== 1.9 = unRAID server Pre-Clear disk /dev/sdg = cycle 1 of 1, partition start on sector 64 = Disk Pre-Clear-Read completed DONE = Step 1 of 10 - Copying zeros to first 2048k bytes DONE = Step 2 of 10 - Copying zeros to remainder of disk to clear it DONE = Step 3 of 10 - Disk is now cleared from MBR onward. DONE = Step 4 of 10 - Clearing MBR bytes for partition 2,3 & 4 DONE = Step 5 of 10 - Clearing MBR code area DONE = Step 6 of 10 - Setting MBR signature bytes DONE = Step 7 of 10 - Setting partition 1 to precleared state DONE = Step 8 of 10 - Notifying kernel we changed the partitioning DONE = Step 9 of 10 - Creating the /dev/disk/by* entries DONE = Step 10 of 10 - Verifying if the MBR is cleared. DONE = Disk Post-Clear-Read completed DONE Disk Temperature: 33C, Elapsed Time: 22:22:06 ========================================================================1.9 == WDC WD15EADS-00S2B0 WD-WCAVY3345713 == Disk /dev/sdg has NOT been precleared successfully == skip=1000 count=200 bs=8225280 returned 04964 instead of 00000 skip=106800 count=200 bs=8225280 returned 06360 instead of 00000 ============================================================================ ** Changed attributes in files: /tmp/smart_start_sdg /tmp/smart_finish_sdg ATTRIBUTE NEW_VAL OLD_VAL FAILURE_THRESHOLD STATUS RAW_VALUE Temperature_Celsius = 119 117 0 ok 33 No SMART attributes are FAILING_NOW 0 sectors were pending re-allocation before the start of the preclear. 0 sectors were pending re-allocation after pre-read in cycle 1 of 1. 0 sectors were pending re-allocation after zero of disk in cycle 1 of 1. 0 sectors are pending re-allocation at the end of the preclear, the number of sectors pending re-allocation did not change. 0 sectors had been re-allocated before the start of the preclear. 0 sectors are re-allocated at the end of the preclear, the number of sectors re-allocated did not change. the full syslog zip file is in my post above. 4gig memory, amd athlon II x3 processor, biostar 785 board...
April 4, 201115 yr Author sorry, what are HBAs ? I'm new around here... My 2nd preclear finished fine, added that to the array and it formatted and added fine. So now I have 1 preclear left that's running. Copying from the unRaid server is still really slow at 171kb/s... Trying to access unMenu I show this in the syslog: Apr 3 23:11:40 Tower unmenu-status: Exiting unmenu web-server, exit status code = 141 Apr 3 23:11:40 Tower unmenu-status: Starting unmenu web-server
April 4, 201115 yr Which Host bus adaptors? pre-clear is known to fail on certain HBA MB combinations. MB SATA ports must be used to pre-clear in this case.
April 4, 201115 yr Which Host bus adaptors? pre-clear is known to fail on certain HBA MB combinations. MB SATA ports must be used to pre-clear in this case. All the symptoms point to a bad memory strip, or one where the voltage timing, and clock speed are not set properly. Have you performed a memory test, preferably overnight. Joe L.
April 4, 201115 yr Author I'm using onboard SATA plus AOC-SASLP-MV8 plus the JMicron eBay special. The HDs that are being cleared are on the SAS and the MB. Oddly enough, the preclear that was on the MB finished the fastest but had errors /dev/sdg (above). The preclear using the SAS finished 2nd but was fine. The 3rd and last preclear is off the MB and finished fine too. no reallocated sectors no errors. All were WD15EADS drives (1.5TB WD Green non AF). I ran memtest for 7 days solid with no memory errors. I'm running a preclear_disk -V on /dev/sdg to perform the post read to see if the errors were a read error or the drive actually didn't get cleared properly. Now that the preclears are finished, the free memory is back up: root@Tower:/boot# free total used free shared buffers cached Mem: 4115664 834112 3281552 0 505024 219536 -/+ buffers/cache: 109552 4006112 Swap: 0 0 0 but copying from unRaid server is still slow... 170kb/s Latest syslog is attached. I'm going to reboot and see what happens. syslog-2011-04-04.zip
April 4, 201115 yr Author I rebooted and I still got very low speeds... 170kb/s. As a diagnostic I checked the network link speed and it was at 100Mb... so I unplugged and replugged the cable into the server and the wall... bingo! back up to 1000Mb link. My reads from the server is now around 55-60Mb/s. I've restarted my torrents and will proceed to preclear my last drive as well as initiate a post-read of the precleared drive above that failed: preclear_disk.sh -V /dev/sdg. Will report tomorrow. Can't believe it could have been the result of a bad network connection!
April 4, 201115 yr Author well... the network speed is consistently high now. Not sure what originally caused the crash in the first place... will keep an eye out on it I guess. I also notiecd that my torrents are disconnected because the unRaid server is losing the link but re-establishing it very quickly... I know I read about this elsewhere so will have to go for a dig. Man, it's kinda frustrating getting this thing to work reliably...
Archived
This topic is now archived and is closed to further replies.