April 15, 201016 yr To run the long tests you'll need to disable the disk spin-down, at unRAID spinning down a disk will abort a "long" test.
April 15, 201016 yr Author Actually something odd just happened in my syslog. Apr 15 11:13:28 Server kernel: mdcmd (194): nocheck Apr 15 11:13:28 Server kernel: md: md_do_sync: got signal, exit... Apr 15 11:13:28 Server kernel: md: recovery thread sync completion status: -4 Apr 15 11:13:31 Server kernel: mdcmd (196): check CORRECT Apr 15 11:13:31 Server kernel: md: recovery thread woken up ... Apr 15 11:13:31 Server kernel: md: recovery thread checking parity... Apr 15 11:13:31 Server kernel: md: using 1152k window, over a total of 1953514552 blocks. Apr 15 11:14:54 Server kernel: mdcmd (226): nocheck Apr 15 11:14:54 Server kernel: md: md_do_sync: got signal, exit... Apr 15 11:14:55 Server kernel: md: recovery thread sync completion status: -4 Apr 15 11:19:15 Server kernel: mdcmd (262): spinup 0 Apr 15 11:19:15 Server kernel: Apr 15 11:19:23 Server kernel: mdcmd (266): spinup 1 Apr 15 11:19:23 Server kernel: Apr 15 11:19:29 Server kernel: mdcmd (270): spinup 2 Apr 15 11:19:29 Server kernel: Apr 15 11:19:35 Server kernel: mdcmd (273): spinup 3 Apr 15 11:19:35 Server kernel: Apr 15 11:19:43 Server kernel: mdcmd (277): spinup 4 Apr 15 11:19:43 Server kernel: Apr 15 11:19:54 Server kernel: mdcmd (281): spinup 5 Apr 15 11:19:54 Server kernel: Apr 15 11:20:06 Server kernel: mdcmd (285): spinup 6 Apr 15 11:20:06 Server kernel: Apr 15 11:20:15 Server kernel: mdcmd (289): spinup 7 Apr 15 11:20:15 Server kernel: Apr 15 11:20:20 Server kernel: mdcmd (293): spinup 8 Apr 15 11:20:20 Server kernel: Apr 15 11:20:26 Server kernel: mdcmd (296): spinup 9 Apr 15 11:20:26 Server kernel: Apr 15 11:20:32 Server kernel: mdcmd (300): spinup 10 Apr 15 11:20:32 Server kernel: Apr 15 11:20:39 Server kernel: mdcmd (304): spinup 0 Apr 15 11:20:39 Server kernel: Apr 15 11:21:26 Server kernel: mdcmd (313): spinup 0 Apr 15 11:21:26 Server kernel: Apr 15 11:21:40 Server kernel: mdcmd (318): spinup 1 Apr 15 11:21:40 Server kernel: Apr 15 11:21:49 Server kernel: mdcmd (321): spinup 2 Apr 15 11:21:49 Server kernel: Apr 15 11:22:01 Server kernel: mdcmd (326): spinup 3 Apr 15 11:22:01 Server kernel: Apr 15 11:22:09 Server kernel: mdcmd (329): spinup 4 Apr 15 11:22:09 Server kernel: Apr 15 11:22:20 Server kernel: mdcmd (334): spinup 5 Apr 15 11:22:20 Server kernel: Apr 15 11:22:31 Server kernel: mdcmd (338): spinup 6 Apr 15 11:22:31 Server kernel: Apr 15 11:22:42 Server kernel: mdcmd (342): spinup 7 Apr 15 11:22:42 Server kernel: Apr 15 11:22:50 Server kernel: mdcmd (346): spinup 8 Apr 15 11:22:50 Server kernel: Apr 15 11:22:57 Server kernel: mdcmd (349): spinup 9 Apr 15 11:22:57 Server kernel: Apr 15 11:23:04 Server kernel: mdcmd (353): spinup 10 Apr 15 11:23:04 Server kernel: No idea what this means... It seems like spinup/spindown is having issues and maybe thats why i'm having issues. As you can see, parity check was started 5 minutes before it attempted to spin up my drives.. twice. The drives were already spun up both times it tried. I did notice the other day that my drives randomly spun up three times and were being read for about 10 minutes for no reason (It showed about 8,000 reads on every data drive on my system when I wasn't even accessing them!). I don't have cache dirs installed and never really figured out why, but I ignored it because it wasn't a big deal. Spindown groups are disabled... but that doesn't look normal. No other errors in syslog. syslog.txt
April 16, 201016 yr Author I just thought of something. I heard that the entire unraid slows down if you fill your hard drives up to much due to the filesystem it uses. Is this to full? Two of my drives have under 10GB free space.
April 16, 201016 yr I just thought of something. I heard that the entire unraid slows down if you fill your hard drives up to much due to the filesystem it uses. Is this to full? Two of my drives have under 10GB free space. That should only affect you if you are copying new files to the array..
April 16, 201016 yr ... No idea what this means... It seems like spinup/spindown is having issues and maybe thats why i'm having issues. As you can see, parity check was started 5 minutes before it attempted to spin up my drives.. twice. The drives were already spun up both times it tried. I did notice the other day that my drives randomly spun up three times and were being read for about 10 minutes for no reason (It showed about 8,000 reads on every data drive on my system when I wasn't even accessing them!). I don't have cache dirs installed and never really figured out why, but I ignored it because it wasn't a big deal. This is exactly the reason why I stopped using unRAID's spinup/spindown feature in the latest versions. You need to bring this bug to Limetech's attention! Email them: [email protected] ...and point them to this thread.
April 16, 201016 yr Author I just wanted to add that it seems like my file transfers are also slow. I was getting 20MB/s earlier.. and now im getting 5MB/s. Transfering files that are roughly 7GB each. I did a disk speed test and all reads were 90MB/s or higher. I'm unsure how to do a write test. I also tried transferring to my computer, and it's still 5MB/s. I'm pretty sure that says it's not a limitation of the hard drives and something else is limiting it in the system. Transfer from my computer to my server results in the server timing out. The network cable is definitely not the issues because this has to be the reason for my slow parity checks. All hard drives passed SMART long test. Real time transfers (over LAN to another computer with known working hard drives). Disk 1 (Onboard): 45MB/s Avg Disk 2 (Onboard): 3MB/s Avg Disk 3 (Onboard): 18MB/s Avg Disk 4 (Onboard): 6MB/s Avg Disk 5 (SATA Card 1): 8MB/s Avg Disk 6 (SATA Card 1): 8MB/s Avg Disk 7 (SATA Card 1): 47MB/s Avg Disk 8 (SATA Card 1): 46MB/s Avg Disk 9 (SATA Card 1): 35MB/s Avg Disk 10 (SATA Card 1): 50MB/s Avg Don't get it... I still can't think of something causing this but unraid, unless all a sudden everything in my system is going faulty.
April 16, 201016 yr Notice that many of the disks which are slow to write are near capacity. Test the interface speed with dd reading from the disk to insure the interface is running at capacity. i.e. hdparm -tT When doing these type of tests, make sure all disks are spun up. Sync the disks. (sync) Drop the cache. (echo 3 > /proc/sys/vm/drop_caches) Then do your test. I would do the test directly on the machine with dd first. Then sync, then drop cache, then over the network. Real time transfers (over LAN to another computer with known working hard drives). Disk 1 (Onboard): 45MB/s Avg Disk 2 (Onboard): 3MB/s Avg 8GB Free Disk 3 (Onboard): 18MB/s Avg 19GB Free Disk 4 (Onboard): 6MB/s Avg 37GB Free Disk 5 (SATA Card 1): 8MB/s Avg 20GB Free Disk 6 (SATA Card 1): 8MB/s Avg 7GB Free Disk 7 (SATA Card 1): 47MB/s Avg 44GB Free Disk 8 (SATA Card 1): 46MB/s Avg 40GB Free Disk 9 (SATA Card 1): 35MB/s Avg 28GB Free Disk 10 (SATA Card 1): 50MB/s Avg 1.6TB Free Example: -- Writing to a disk in the protected array: dd if=/dev/zero bs=1M count=1000 of=/mnt/disk2/test.dd 1048576000 bytes (1.0 GB) copied, 42.68 s, 24.6 MB/s rm -v /mnt/disk2/test.dd I usually use 4 to 5 GB to simulate the size of a DVD and use all available memory. You will notice something on the near capacity disks. The disk will be very busy during the first part of the write. Then in a few moments you will see activity on the parity. That is when the real writes start to occur. This is not a problem with unRAID. It's a by product of near capacity disk on reiserfs. If you have 5400 drives, this situation is exasperated. It's even worse over a network with samba, because samba connections can time out. This slow write speed has nothing to do with the parity check/generate speed. This is my writeread test script. I use this so I can watch the performance as it occurs with dd. I should have probably added the drop cache line. You can add it if you like. echo 3 > /proc/sys/vm/drop_caches #!/bin/bash if [ -z "$1" ] then echo "Usage: $0 outputfilename" exit fi if [ -f "$1" ] then echo "removing: $1" rm -vf $1 sync fi bs=1024 count=10000000 count=4000000 total=$(( $bs * $count)) echo "writing $total bytes to: $1" dd if=/dev/zero bs=$bs count=$count of=$1 & BGPID=$! sleep 5 while ps --no-heading -fp $BGPID >/dev/null do kill -USR1 $BGPID sleep 5 done echo "write complete, syncing" sync echo "reading from: $1" dd if=$1 bs=$bs count=$count of=/dev/null echo "removing: $1" rm -vf $1
April 16, 201016 yr Here's something i've been wondering: My motherboard allows me to set my PCI-X slots to 100mhz or 133mhz. Right now they are both on AUTO. Can I just set them box to 133MHz even though the bottom one is rated for 100MHz? I'm just confused on why they would allow you to overclock the PCI-X Slot.. seems risky. I would set them manually for their rated speeds. Or even try a lower speed just to test. Peter
April 16, 201016 yr Author Notice that many of the disks which are slow to write are near capacity. Test the interface speed with dd reading from the disk to insure the interface is running at capacity. i.e. hdparm -tT When doing these type of tests, make sure all disks are spun up. Sync the disks. (sync) Drop the cache. (echo 3 > /proc/sys/vm/drop_caches) Then do your test. I would do the test directly on the machine with dd first. Then sync, then drop cache, then over the network. Real time transfers (over LAN to another computer with known working hard drives). Disk 1 (Onboard): 45MB/s Avg Disk 2 (Onboard): 3MB/s Avg 8GB Free Disk 3 (Onboard): 18MB/s Avg 19GB Free Disk 4 (Onboard): 6MB/s Avg 37GB Free Disk 5 (SATA Card 1): 8MB/s Avg 20GB Free Disk 6 (SATA Card 1): 8MB/s Avg 7GB Free Disk 7 (SATA Card 1): 47MB/s Avg 44GB Free Disk 8 (SATA Card 1): 46MB/s Avg 40GB Free Disk 9 (SATA Card 1): 35MB/s Avg 28GB Free Disk 10 (SATA Card 1): 50MB/s Avg 1.6TB Free Example: -- Writing to a disk in the protected array: dd if=/dev/zero bs=1M count=1000 of=/mnt/disk2/test.dd 1048576000 bytes (1.0 GB) copied, 42.68 s, 24.6 MB/s rm -v /mnt/disk2/test.dd I usually use 4 to 5 GB to simulate the size of a DVD and use all available memory. You will notice something on the near capacity disks. The disk will be very busy during the first part of the write. Then in a few moments you will see activity on the parity. That is when the real writes start to occur. This is not a problem with unRAID. It's a by product of near capacity disk on reiserfs. If you have 5400 drives, this situation is exasperated. It's even worse over a network with samba, because samba connections can time out. This slow write speed has nothing to do with the parity check/generate speed. This is my writeread test script. I use this so I can watch the performance as it occurs with dd. I should have probably added the drop cache line. You can add it if you like. echo 3 > /proc/sys/vm/drop_caches #!/bin/bash if [ -z "$1" ] then echo "Usage: $0 outputfilename" exit fi if [ -f "$1" ] then echo "removing: $1" rm -vf $1 sync fi bs=1024 count=10000000 count=4000000 total=$(( $bs * $count)) echo "writing $total bytes to: $1" dd if=/dev/zero bs=$bs count=$count of=$1 & BGPID=$! sleep 5 while ps --no-heading -fp $BGPID >/dev/null do kill -USR1 $BGPID sleep 5 done echo "write complete, syncing" sync echo "reading from: $1" dd if=$1 bs=$bs count=$count of=/dev/null echo "removing: $1" rm -vf $1 I honestly wish I understood your post more than I do. I have almost 0 knowledge of linux and the advanced things of unRAID. What I did understand is that I should free up more space on some of my hard drives. It seems I need roughly 40GB free for them, so I will move some of the data off of a few of them down to another hard drive.
April 16, 201016 yr I honestly wish I understood your post more than I do. I have almost 0 knowledge of linux and the advanced things of unRAID. What I did understand is that I should free up more space on some of my hard drives. It seems I need roughly 40GB free for them, so I will move some of the data off of a few of them down to another hard drive. No, you do not have to free space up if you do not plan to do time dependent writes. If you are done filling the drive, then that's it. If you fill it a lil more, it will take time. Sorry if my notes are skimming above your knowledge level. I'm sure someone will help elaborate. No fret, you will get it all eventually.
April 16, 201016 yr the writeread script I have in the other post is a better way of testing write and read speeds to your array locally on unRAID Telnet in as root. mkdir -p /boot/custom/bin Put the script I've included into that directory. and run the script on each drive as in /boot/custom/bin/writeread /mnt/disk1/test.dd This will help you know if there is a speed issue with the drive itself rather then going through the lan.
April 16, 201016 yr Author Well what I don't understand is why some of those discs are so slow at transfering files to another computer. It makes it impossible for me to transfer things off my server to other computers, which I do regularly. I freed up 70GB on disc 2, and it still transfers to my main computer at around 8MB/s.. whereas the disk8-10 transfer at 45+MB/s. I simply cannot have speeds that slow for transfers over the network. If I do a "Disk Speed Test" using unMENU, all my data drives read at 90+MB/s. Transfering files off of a hard drive and onto another computer should only require read speeds on the drive that the data is coming from, so I don't get this.
April 16, 201016 yr free space is only an issue for writes when the drive is near capacity. I don't think the unmenu speed test exercises the drives the same way as the dd would. That's something for others to answer. 8MB/s is too slow. I agree. I don't have all the answers on this one.
April 16, 201016 yr I peeked at your syslog, I did not see anything that jumped out and said something is wrong.
April 16, 201016 yr Author I peeked at your syslog, I did not see anything that jumped out and said something is wrong. Tell me if this makes sense to you... I went into my BIOS, and thought about what exactly I changed since the day it stopped working. The only thing I did was "exclude" my SATA cards from the boot order menu. So I re-added them. Parity Sync at 82MB/s now.. I'm not 100% sure this was the problem, but it is now working and it wasn't a second ago. I don't get how not having the SATA cards in the boot order menu would cause this, since it boots from the first thing I have set in there, the USB stick. I'll do some more testing, but I may have fixed it. Also, after doing this my transfer rates to my computer drastically improved as well, disk 2 is now going 44MB/s compared to 8MB/s. I'd also like to add that disk 2 is on onboard, and isn't even on the SATA cards. So once again, this makes no sense to me.. but i'm glad it's working now and I hope it doesn't start screwing up again. EDIT: Confirmed it, removed them from the boot menu order again.. back to the same issues, re-added them and it's fine. Thanks everyone for the help, I still don't get why it does this and it doesn't make sense to me but atleast it's working (for now). I wouldn't be surprised if it starts doing it again though.
April 16, 201016 yr Sounds very messed up to me too. Maybe the SATA cards don't get properly initialized unless they're in the Boot Order?
April 17, 201016 yr Can you try to (dis)prove that theory? Go back to BIOS and remove SATA from the Boot Order. See if the problem returns. It will be interesting either way.
April 17, 201016 yr Perhaps an initialization issue in bios. This was an odd one. Yeah! Just when you thought that you've heard them all!
April 17, 201016 yr Author Can you try to (dis)prove that theory? Go back to BIOS and remove SATA from the Boot Order. See if the problem returns. It will be interesting either way. Yes, I did that. Problem returned when I did.
April 17, 201016 yr Can you try to (dis)prove that theory? Go back to BIOS and remove SATA from the Boot Order. See if the problem returns. It will be interesting either way. Yes, I did that. Problem returned when I did. unfrigginbelievable!
April 17, 201016 yr Author Can you try to (dis)prove that theory? Go back to BIOS and remove SATA from the Boot Order. See if the problem returns. It will be interesting either way. Yes, I did that. Problem returned when I did. unfrigginbelievable! Thats what I said.. it makes no sense! Oh well, i'm glad it works now.. and is still working. I'll post back if it happens again, which wouldn't be to surprising, because this 'fix' makes no sense.
April 17, 201016 yr Thats what I said.. it makes no sense! Oh well, i'm glad it works now.. and is still working. I'll post back if it happens again, which wouldn't be to surprising, because this 'fix' makes no sense. For some reason that reminded me about the "ultimate" new car that was designed by m$oft.. In the owner's manual you could find... Should the engine start acting up, do the following: 1. Pull over to the side of the road; 2. Shut off the engine; 3. Get out of the car; 4. Get back in the car; 5. Start the engine. The above steps should fix the problem.
Archived
This topic is now archived and is closed to further replies.