April 20, 201016 yr Apr 19 16:53:18 lime kernel: ata2: hard resetting link Apr 19 16:53:18 lime kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata3: hard resetting link Apr 19 16:53:18 lime kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata4: hard resetting link Apr 19 16:53:18 lime kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata6: hard resetting link Apr 19 16:53:18 lime kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata7: hard resetting link Apr 19 16:53:18 lime kernel: ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata8: hard resetting link That doesn't look good! I'll let others answer that. If this is within moments after a reboot, it's normal. If it's after being up for a while, then there are issues.
April 20, 201016 yr That is definitely wrong. Resets are generally associated with pauses, typically 5 seconds each with drive exceptions, although the ones above are happening in parallel. It appears you have something systemically wrong. Please zip up and post your syslogs, for someone to advise.
April 20, 201016 yr I get these when I power up from a cold power state. The drives are in standby mode which is something I set in software or jumper. When the kernel starts up the drives, I see these messages. If the system has been up for a while and a spindown command or spin up command, sometimes I see these messages. Although, There is usually a pause of a few seconds. Most of the time the drive recovers, other times drives go offline. As suggested, posting the complete syslog is needed to see events prior and after.
April 20, 201016 yr Author Apr 19 16:53:18 lime kernel: ata2: hard resetting link Apr 19 16:53:18 lime kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata3: hard resetting link Apr 19 16:53:18 lime kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata4: hard resetting link Apr 19 16:53:18 lime kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata6: hard resetting link Apr 19 16:53:18 lime kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata7: hard resetting link Apr 19 16:53:18 lime kernel: ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata8: hard resetting link That doesn't look good! I'll let others answer that. If this is within moments after a reboot, it's normal. If it's after being up for a while, then there are issues. Its from boot, after swapping drive 10 for a larger one. Are there any known issues with the RB-1200 as delivered? I haven't cracked the case, apart from the drives its as delivered from lime-technology, I am a little skeptical of the Power Supply at only 550w
April 20, 201016 yr Author I get these when I power up from a cold power state. The drives are in standby mode which is something I set in software or jumper. When the kernel starts up the drives, I see these messages. If the system has been up for a while and a spindown command or spin up command, sometimes I see these messages. Although, There is usually a pause of a few seconds. Most of the time the drive recovers, other times drives go offline. As suggested, posting the complete syslog is needed to see events prior and after. The start of that syslog is attached.. I just chopped off the end which is full of duplicate object messages (which I'll look into after posting this) and mover messages lime-syslog-post.txt
April 20, 201016 yr The start of that syslog is attached.. I just chopped off the end which is full of ... If you just zip it then you won't need to chop off anything.
April 20, 201016 yr Author The start of that syslog is attached.. I just chopped off the end which is full of ... If you just zip it then you won't need to chop off anything. I chopped it off out of privacy concerns, not size.
April 20, 201016 yr The start of that syslog is attached.. I just chopped off the end which is full of ... If you just zip it then you won't need to chop off anything. I chopped it off out of privacy concerns, not size. Oh, the porn. Understandable.
April 20, 201016 yr Author The start of that syslog is attached.. I just chopped off the end which is full of ... If you just zip it then you won't need to chop off anything. I chopped it off out of privacy concerns, not size. Oh, the porn. Understandable. Not just porn (That's what the equal logic is for, priorities!) The Dupe File thing isn't making sense.. The files don't seem to actually be dupes (either via the Dupe Files script or crawling the directories, interesting).
April 20, 201016 yr I should know better than to make snap judgments based on small excerpts of a syslog. Those resets appear to be normal for your setup, as they are a part of the setup of most of the drives attached to your Promise cards. They are odd, not normal, as they are also accompanied by raising an exception and a hotplug status message, but after the hard reset, the drives appear to perform correctly without any further issues. Below is a typical setup of one of your Promise-connected drives: Apr 19 16:53:18 lime kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata2.00: ATA-8: WDC WD10EACS-00ZJB0, 01.01B01, max UDMA/133 Apr 19 16:53:18 lime kernel: ata2.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 0/32) Apr 19 16:53:18 lime kernel: ata2.00: configured for UDMA/133 Apr 19 16:53:18 lime kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0xf t4 Apr 19 16:53:18 lime kernel: ata2: hotplug_status 0x22 Apr 19 16:53:18 lime kernel: ata2: hard resetting link Apr 19 16:53:18 lime kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Apr 19 16:53:18 lime kernel: ata2.00: configured for UDMA/133 Apr 19 16:53:18 lime kernel: ata2: EH complete Apr 19 16:53:18 lime kernel: scsi 1:0:0:0: Direct-Access ATA WDC WD10EACS-00Z 01.0 PQ: 0 ANSI: 5 Apr 19 16:53:18 lime kernel: sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Apr 19 16:53:18 lime kernel: sd 1:0:0:0: [sdb] Write Protect is off Apr 19 16:53:18 lime kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Apr 19 16:53:18 lime kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Apr 19 16:53:18 lime kernel: sdb: sdb1 Apr 19 16:53:18 lime kernel: sd 1:0:0:0: [sdb] Attached SCSI disk I would not do anything to your array until the rebuild of Disk 10 safely completes, but afterward, check to see if new messages appear in the syslog after one of those writes with delays.
April 20, 201016 yr Author cd to your cache drive and run a dd speed test. dd if=/dev/zero of=test.dd bs=1024 count=4096000 root@lime:/mnt/cache# dd if=/dev/zero of=test.dd bs=1024 count=4096000 4096000+0 records in 4096000+0 records out 4194304000 bytes (4.2 GB) copied, 51.3371 s, 81.7 MB/s (The rebuild has finished)
April 20, 201016 yr cd to your cache drive and run a dd speed test. dd if=/dev/zero of=test.dd bs=1024 count=4096000 root@lime:/mnt/cache# dd if=/dev/zero of=test.dd bs=1024 count=4096000 4096000+0 records in 4096000+0 records out 4194304000 bytes (4.2 GB) copied, 51.3371 s, 81.7 MB/s (The rebuild has finished) This is the correct speed. I get the same on my F3 1TB drive. If the rebuild has completed I suppose you could do another test transfer to the cache. or start off with a reboot, do the test and provide that syslog for review. Just grep -v the dup messages.
April 20, 201016 yr Author This is the correct speed. I get the same on my F3 1TB drive. If the rebuild has completed I suppose you could do another test transfer to the cache. or start off with a reboot, do the test and provide that syslog for review. Just grep -v the dup messages. Will do, will have to wait 'til tonight... back to work.
April 20, 201016 yr Author While Sending: Bad Worse (copy STILL in process) Top on lime (while dieing) While Receiving: Typical Bad Top on lime (bad) I have not had a chance to try any of the other remedies offered, I'll investigate and share any findings. A big thanks to everyone for the help!
April 21, 201016 yr When copies fail at the start, I look at reiserfs allocation delay. This is prominent with 5400 RPM drives and a near capacity drive. I'm getting this, what is the solution? How do you change the reiserfs allocation delay?
April 21, 201016 yr When copies fail at the start, I look at reiserfs allocation delay. This is prominent with 5400 RPM drives and a near capacity drive. I'm getting this, what is the solution? How do you change the reiserfs allocation delay? I've only seen this on "large" drives that are near capacity. What I've done in the past is use rsync or copy it to folder on a drive less full, then use rsync to move the files with rsync -avP --remove-sent-files sourcefiles destinationdirectory This is like doing a move, only it's more efficient. If I do not want to blow out the buffer cache because I am doing something else then I use --bwlimit=8192 to keep the copy within reasonable limits. But this is my method.
April 21, 201016 yr Thanks, I was almost convinced it was my drive, but the symptoms were so strange. Using TeraCopy it would fail a file copy the first time and then the second time it would work. BTW this is on a 2T. 5400 WD that was 97% full. Sorry to step on aobrien's thread, I'm anxious to see the solution to his problem.
April 21, 201016 yr Author One More tidbit for today, wonder if this has anything to do with my problem /dev/sda: Timing cached reads: 2180 MB in 2.00 seconds = 1090.11 MB/sec Timing buffered disk reads: 300 MB in 3.02 seconds = 99.37 MB/sec /dev/sdb: Timing cached reads: 2128 MB in 2.00 seconds = 1063.88 MB/sec Timing buffered disk reads: 226 MB in 3.02 seconds = 74.95 MB/sec /dev/sdc: Timing cached reads: 2130 MB in 2.00 seconds = 1065.36 MB/sec Timing buffered disk reads: 266 MB in 3.01 seconds = 88.48 MB/sec /dev/sdd: Timing cached reads: 2146 MB in 2.00 seconds = 1073.40 MB/sec Timing buffered disk reads: 302 MB in 3.01 seconds = 100.46 MB/sec /dev/sde: Timing cached reads: 2146 MB in 2.00 seconds = 1073.27 MB/sec Timing buffered disk reads: 234 MB in 3.01 seconds = 77.82 MB/sec /dev/sdf: Timing cached reads: 1944 MB in 2.00 seconds = 972.15 MB/sec Timing buffered disk reads: 30 MB in 3.20 seconds = [color=red][b]9.38 MB/sec[/b][/color] /dev/sdg: Timing cached reads: 2140 MB in 2.00 seconds = 1069.76 MB/sec Timing buffered disk reads: 218 MB in 3.00 seconds = 72.64 MB/sec /dev/sdh: Timing cached reads: 2142 MB in 2.00 seconds = 1071.44 MB/sec Timing buffered disk reads: 308 MB in 3.00 seconds = 102.50 MB/sec /dev/sdi: Timing cached reads: 2150 MB in 2.00 seconds = 1074.98 MB/sec Timing buffered disk reads: 260 MB in 3.00 seconds = 86.60 MB/sec /dev/sdj: Timing cached reads: 2144 MB in 2.00 seconds = 1072.16 MB/sec Timing buffered disk reads: 360 MB in 3.00 seconds = 119.98 MB/sec /dev/sdk: Timing cached reads: 2140 MB in 2.00 seconds = 1070.33 MB/sec Timing buffered disk reads: 326 MB in 3.01 seconds = 108.44 MB/sec /dev/sdl: Timing cached reads: 2148 MB in 2.00 seconds = 1074.34 MB/sec Timing buffered disk reads: 350 MB in 3.00 seconds = 116.51 MB/sec /dev/sdm: Timing cached reads: 2144 MB in 2.00 seconds = 1072.32 MB/sec Timing buffered disk reads: 278 MB in 3.01 seconds = 92.42 MB/sec nevermind sdf is the flash drive... but a couple of those others are a little pokey too. Because they're WDC GreenPower.. Will work on swapping them out..
April 21, 201016 yr One More tidbit for today, wonder if this has anything to do with my problem /dev/sdf: Timing cached reads: 1944 MB in 2.00 seconds = 972.15 MB/sec Timing buffered disk reads: 30 MB in 3.20 seconds = [color=red][b]9.38 MB/sec[/b][/color] sdf is your flash key.
April 22, 201016 yr Author Well, very little progress (work's been busy), starting with purko's tweaks (and tweaking them little) I've improved things some. But I think I also have a bad drive now, Twice on restart one of those #@$ Green Powers didn't come up.. I'll have to get a replacement, swap it out, and rebuild before I test further, if the problem even continues. The thing that still puzzles me, is why it happened at the end of each file write, and btw this seems to be less true now, but I haven't really been testing for that..
April 22, 201016 yr But I think I also have a bad drive now, Twice on restart one of those #@$ Green Powers didn't come up.. I'll have to get a replacement, swap it out, and rebuild before I test further, if the problem even continues. Review the SMART logs, Do a SMART short and long tests, then review the logs again. If there are any issues with the power supply, it will affect the spinup of drives.
April 22, 201016 yr The thing that still puzzles me, is why it happened at the end of each file write, and btw this seems to be less true now, but I haven't really been testing for that.. Sounds like the buffer cache being flushed.
April 27, 201016 yr Author Smart reports turned up nothing wrong with that dirve.. I'm still going to swap it out but I'm in no hurry Sounds like the buffer cache being flushed. The problem persists, I'm not even sure it really has gotten better, it seems like it may even be worse. My go file (thanks to purko for the tips) looks like this: root@lime:~# cat /boot/config/go #!/bin/bash logger -t "$0[$$]" "##### Starting the GO script... #####" # First, a few tweaks... (found to improve things on my particular box) for i in /sys/block/[hs]d? ; do echo 128 > $i/queue/max_sectors_kb ; done 2>/dev/null for i in /sys/block/[hs]d? ; do echo cfq > $i/queue/scheduler ; done 2>/dev/null sysctl -w vm.min_free_kbytes=8192 # sl:2497 sysctl -w vm.dirty_expire_centisecs=900 # sl:3000 tm:100 sysctl -w vm.dirty_writeback_centisecs=300 # sl:500 tm:50 sysctl -w vm.dirty_ratio=20 # sl:10 tm:10 sysctl -w vm.dirty_background_ratio=10 # sl:5 tm:5 # fix the FTP server mask for uploaded files echo "anon_umask=0022" >> /etc/vsftpd.conf # Start the Management Utility /usr/local/sbin/emhttp & /boot/unmenu/uu # reload our crontabs sleep 60 crontab /mnt/user/aob/lime_backup/crontab I'm getting more frustrated with this, today copying some 40 files totalling ~20GB is going to take an hour because of the pauses..
April 27, 201016 yr Author top - 17:20:40 up 4 days, 23:52, 1 user, load average: 0.35, 0.46, 0.44 Tasks: 97 total, 1 running, 96 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 2073560k total, 2013020k used, 60540k free, 45996k buffers Swap: 0k total, 0k used, 0k free, 1791796k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 776 304 264 S 0 0.0 0:01.87 init 2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0 While in a "pause" it just seems to sti there like a lump doing nothing...
Archived
This topic is now archived and is closed to further replies.