WeeboTech

I had a script called write4gb which is similiar to #!/bin/bash bs=1024 count=4000000 # count=10000000 dd if=/dev/zero bs=$bs count=$count of=$1 ideally you want it to be 2x ram so I had the write10gb script. #!/bin/bash bs=1024 # count=4000000 count=10000000 dd if=/dev/zero bs=$bs count=$count of=$1 you just run it with ./write4gb /mnt/disk1/test.dd Over here I have my writeread10gb script. Writes 10gb, then reads it back. https://code.google.com/p/unraid-weebotech/downloads/detail?name=writeread10gb&can=2&q=#makechanges You can cannibalize it and use only the write part of it. What I like about it is that it will give you a status per interval so you can see how buffer cache and kernel tuning come into play. Throughout the board I mention that I can burst at high speed. This is because of buffer caching and some tunings. I'm able to get upwards of 60MB/s sometimes higher depending on the drive. Burst to about 500GB, then is slows down. For me, this is what I need as I move allot of MP3's around, tag the, add artwork. etc. etc. So I want to burst write as fast as I can even if it's only for a smaller subset of data that I may move. i.e. movies. In any case, fastest drives on parity and the data disk you are testing matter. Plus, as empty of a filesystem as possible.

I found the best way to test writes is eliminate some of the IO by reading /dev/zero quick scriptlet. bs=1024 count=4000000 count=10000000 dd if=/dev/zero bs=$bs count=$count of=$1 One issue when testing writes is you want as empty of a file system as possible. Allocation of the directory and file nodes takes time. Searching through the filesystem looking for free blocks takes time. Remove the file before your next test or it will be skewed. OR Create the file, then overwrite the same file each time, this way the allocation is already done.

If the drives are capable of doing 160MB/s and work is progressing, ie. parity check is moving forward while other work is operating, it's not thrashing. If I'm getting the maximum possible performance out of the drives and work is progressing, it's not thrashing. From a parity check, it's performing, from a user perspective, it's performing slowly. Competing. What you define as 'excessive seeks' is incorrect. IO wait in a queue, is not thrashing. I think we are all describing the same thing, just using different words. Semantics. I'm using the term thrashing to describe the behavior of disk heads inefficiently moving around to different cylinders/sectors to satisfy competing requests. In all my years of IT, I've never heard the term thrashing used to describe a stall, but perhaps I've just never heard the correct technical definition. I think we all agree work is getting done, it's just inefficient due to competing resources. This is where load prioritization in unRAID, allowing you to put one queue at a higher priority over another, would be the best possible solution. Work still gets done, but in the order the user prefers, and the work becomes somewhat more efficient since there will be less head movement to satisfy competing requests, as complementary requests with the same priority get bundled together. Thrashing by it's technical term is in dealing with virtual memory and paging. In our usage, it's by competition of resources to the point that the workload has grown so great that no work can be done efficiently. But technically, that's not the correct definition of the term. If I am reading a disk by 1 process and getting 90MBs locally. Then by 2 processes I get 40MB/s. Then by 3 I get further divided at a somewhat equal level, i.e. 30MB/s, It's not thrashing until the point that every process suffers so much that it's almost pointless to continue.

I've found that rebuilds or generates to be faster then read checks. It could also be the direction of the traffic on the bus or controller. The PCIe bus is full duplex. It could also be that write caching on the hardware comes into play. When I used my Areca with write caching my parity generate speed improvement was measurable.

What is being described is not "thrashing". It's really competition for resources, but not thrashing. The drives are not being overworked to the point that nothing is getting done. Each new access will cause the heads to move, thus causing each job to run slower. When the drives are overworked where absolutely nothing is getting done on the system, because the system is so busy doing housekeeping, that is thrashing. By balancing out these tunables, you can alleviate some of the competition and starvation of one type of access with another. FWIW, on my 4 drive micro server system, my values are Tunable (md_num_stripes): 4096 Tunable (md_write_limit): 2048 Tunable (md_sync_window): 1280 I'm doing a parity generate (not just a check) to a 4TB parity drive. I'm also copying a file from disk1 to another server at from 60MB/s with Win 7. At the same very minute, I have the following stats. Total size: 4 TB Current position: 749.03 GB (19%) Estimated speed: 156.88 MB/sec Estimated finish: 345 minutes I would hardly call this thrashing. Writes will probably drop to about 10-20MB/s as expected. Still, this is not thrashing. The heads are moving, the drives are doing what they are designed to do, and work is still getting done. Albeit, divided among the tasks.

I had an array filled with these WD10EACS. Slowest 1TB drives I've owned. They were fine for storing and reading movies, but anything else was painfully slow. Couple these with the PCI card and that's a bottleneck(PCI) and a half(slower drives). The maximum speed you can achieve with that PCI card is 133MB/s. 266MB/s if on PCI 2.0. You would benefit greatly from a PCIe controller.

You can also try with less physical ram or the mem=4095 parameter to see if that helps.

add the redirect part of the commands echo "# Run a Parity Check on the First Monday of each Month at 12:00am" >> /tmp/crontab echo "0 0 1-7 * * test $(date +%u) -eq 1 && /root/mdcmd check CORRECT" >> /tmp/crontab or via here document with cat (can also be done with echo.) cat <<-EOF >> /tmp/crontab # Run unRAID Fan Speed script every 5 minutes */5 * * * * /boot/unraid-fan-speed.sh >/dev/null # Run a Parity Check on the First Monday of each Month at 12:00am 0 0 1-7 * * test $(date +%u) -eq 1 && /root/mdcmd check CORRECT EOF

This works great for regular writes to the filesystems. I'm not sure it comes into play for the party check. I believe the md buffers are in low memory. At least that was my experience when tuning them years ago.

When expanding these values to very high limits, I found regular reads/writes alone would not create an OOM condition. I would have to do that while running a full "deep" find / -ls >/dev/null down the whole array. It depends on how many files you have on the filesystem. cache_dirs will also help reveal a problem if you have a large number of files.

I like the idea allot. I sometimes mistake which server I'm on.

There's a whole level of virtualization going on there with FUSE. What is the difference in CPU between the two machines? What are the bogomips on the machine(s) you are comparing. How much ram is in each machine? do you have the himem is dirtayable option set?

What is the model of your drive? How are you benchmarking the write speeds?

That was the idea I had a while back. BTW there is a particular syba card that allows 4 ports on a PCIe x1 slot. It has 2 eSATA ports and 2 internal SATA ports. It's no longer available on the regular market. I found 2 on the secondary market. It worked for me. It wouldn't be the best performer, but it sure would max out ports !!! As far as the opening, there's enough room in backup of the upper metal to drill and nibble space for eSATA ports. You would need to evacuate parts to insure no shavings fall on the motherboard, but it could give you 6 or 8 ports from the x4/x1 slot.

But during a parity check or drive rebuild, it will still be accessing all the drives at the same time. Accessing one drive at a time is no problem, like when writing data to the array. I tried staggering the drives in my first unRAID setup but the difference in my use was minimal. I ended up using a cache drive with that setup. With my second unRAid setup I put all the drives in order of each enclosure. I'm not using a cache drive in that setup and have been pleased with the speed results. While the driver attempts to read the drives, I think it has to read each drive sequentially. So by ordering the drives to access the channel once every round vs accessing the channel 4 times one behind the other may alleviate contention for the channel. There was someone who did this years ago when port multipliers were new and he said it gave him a slight increase in speed. YMMV. It may not be worth it considering how you have to arrange the drives. I know when I tested PMP support using 4 drives simultaneously, I got something like 15MB/s per drive. In comparison using the silicon image, 1 drive was 120MB/s 2 drives were 60MB/s, 3 drives were 40MB/s and 4 drives were 30MB/s. There is still the option of putting the 2 eSATA brackets ports on the back of the unit after it is nibbled out. You may be able to fit 4 there. I haven't measured it. That's what I did with my old chenbro chassis years back.

The other choice is to access the drives in each box in a round robin fashion so you are not accessing drives sequentially. That has it's own downside in management and maintenance. I've noticed the same issues with various vendors and how they handle port multiplier access.

What is being described here, is not necessarily thrashing i.e. until Rob J's backup starts to interfere with the ability of what cache_dirs is doing. In my system, cache_dirs created an environment that was wildly out of control and useless. Now that's thrashing. What may serve everyone with various needs is to have cache_dirs check for the existence of a flag file. If this flag file exists, the find is skipped until the next interval. If this flag file's size is greater then 0, read it and use it as an argument to sleep. This way we can set an estimate of how long it should pause. i.e. here is some example scriplets. #!/bin/bash [ ${DEBUG:=0} -gt 0 ] && set -x -v FLAGFILE=/tmp/cache_dirs.sleepflag NORMAL_INTERVAL=${NORMAL_INTERVAL:=60} if [ -s ${FLAGFILE} ] then sleep $(<${FLAGFILE}) else if [ -e ${FLAGFILE} ] then sleep ${NORMAL_INTERVAL} fi fi if flag exists with no size. root@rgclws:/home/rcotrone/sh # DEBUG=3 NORMAL_INTERVAL=10 ./sleepflag.sh FLAGFILE=/tmp/cache_dirs.sleepflag + FLAGFILE=/tmp/cache_dirs.sleepflag NORMAL_INTERVAL=${NORMAL_INTERVAL:=60} + NORMAL_INTERVAL=10 if [ -s ${FLAGFILE} ] then sleep $(<${FLAGFILE}) else if [ -e ${FLAGFILE} ] then sleep ${NORMAL_INTERVAL} fi fi + '[' -s /tmp/cache_dirs.sleepflag ']' + '[' -e /tmp/cache_dirs.sleepflag ']' + sleep 10 if flag exists with a size root@rgclws:/home/rcotrone/sh # echo 15 > /tmp/cache_dirs.sleepflag root@rgclws:/home/rcotrone/sh # DEBUG=3 NORMAL_INTERVAL=10 ./sleepflag.sh FLAGFILE=/tmp/cache_dirs.sleepflag + FLAGFILE=/tmp/cache_dirs.sleepflag NORMAL_INTERVAL=${NORMAL_INTERVAL:=60} + NORMAL_INTERVAL=10 if [ -s ${FLAGFILE} ] then sleep $(<${FLAGFILE}) else if [ -e ${FLAGFILE} ] then sleep ${NORMAL_INTERVAL} fi fi + '[' -s /tmp/cache_dirs.sleepflag ']' <${FLAGFILE}) <${FLAGFILE}) <${FLAGFILE} + sleep 15 So now rather then forcing cache_dirs to calculate the wait time. "For now" we can have the presence of a flag do that. Later on, we can have something else that monitors parity status drop this flag somewhere and/or possibly fill it in with a value. It would be good for backups to drop a known duration into this file also.

"Thrash and cache" is an untrue statement now that two people have proven that there is really no thrashing going on. In fact, it's doing exactly what it was designed to do. Walk through every single dentry in the kernel's tables to keep the entry from being expired.

I had the following distribution of files in a system with 8GB of ram. root@unRAID:/mnt/disk1/cache/flocate# wc -l disk*.filelist 146485 disk1.filelist 4979 disk10.filelist 4270 disk11.filelist 3013 disk12.filelist 2377 disk13.filelist 2530 disk14.filelist 63326 disk15.filelist 4761906 disk2.filelist 7141 disk3.filelist 2704215 disk4.filelist 169797 disk5.filelist 0 disk6.filelist 610 disk7.filelist 2647 disk8.filelist 1601 disk9.filelist 7874897 total I don't remember the default for vfs_cache_pressure. I thought cache_dirs changed it to 100 to keep dentries around.

There are a number of different buffers being used here. The buffer cache buffers filesystem and file data. The dentry cache buffers directory entry filesystem data. The md/unraid buffers. From what I know the dentry and md/unraid buffers are in low memory. The do not normally impact one another. If there is no cache pressure, then the dentry cache will exist until the dentry expires. Normally you wont see an impact unless you have so many files (like I did) that they cannot all be stored in the dentry cache. Then the dentry cache starts dropping entries to hold newer entries. As this happens you start reading directories from the disk instead of the hash table. That's when it could impact parity checks.

I thought there was a value from the /proc/mdcmd that had a time value in seconds. mdResync=3907018532 mdResyncCorr=1 mdResyncPos=4265472 mdResyncDt=30 mdResyncDb=4265472 /* time delta in seconds */ dt = ((jiffies - mddev->resync_mark) / HZ); if (!dt) dt++; p += sprintf(p, "mdResyncDt=%lu\n", dt); /* resync'ed blocks delta */ db = resync - (mddev->resync_mark_cnt/2); p += sprintf(p, "mdResyncDb=%llu\n", db); /* tmm: following code is omitted because it requires 64-bit division * which requires use of do_div() and is a pain-in-the-neck to use, * instead we output the two deltas above so that user space can do * whatever cacluation it wants. * * We do not want to overflow, so the order of operands and * the * 100 / 100 trick are important. We do a +1 to be * safe against division by zero. We only estimate anyway. * * dt: time from mark until now * db: blocks written from mark until now * rt: remaining time */ // unsigned long long max_blocks = mddev->recovery_running/2; // unsigned long long res; // unsigned long long rt; /* compute completion percentage */ // res = resync / (max_blocks/1000 + 1); // p += sprintf(p, "mdResyncPrcnt=%llu.%llu\n", res/10, res % 10); /* sync rate in blocks/sec */ // p += sprintf(p, "mdResyncSpeed=%llu\n", db/dt); /* compute remaining time in minutes */ // rt = (dt * ((max_blocks-resync) / (db/100+1)))/100; // p += sprintf(p, "mdResyncFinish=%llu.%llu\n", rt / 60, (rt % 60)/6); } else { p += sprintf(p, "mdResyncPos=0\n"); p += sprintf(p, "mdResyncDt=0\n"); p += sprintf(p, "mdResyncDb=0\n"); } Once you determine a lengthy process is executing, change the frequency of the loop, Expand it by minutes, tens of minutes, etc, etc.

I'm not saying that quiescing cache_dirs is a bad idea, nor preventing it from quiescing a bad idea. I think the option should exist for people who need it. In one case, a user reports it slows down full array parity operations. Another user reports, it makes very little different. I can say from experience, cache_dirs caused problems on my array. Not because of a programming issue, just the sheer magnitude of files I had on my array. So point is, It would be helpful if it is programmed as an option for those who need it or choose to use it.

The bios needs to be adjusted for sata port 5,6. In bios under Chipset / SouthBridge Configuration / SB Sata Configuration Disable "SATA IDE Combined Mode" - sets port 5 and 6 to use AHCI There might be other settings, that's what I could remember.

The buffers are in low memory. Even if you have a large amount of ram, you can still have pressure in low memory. In addition there is vfs_cache_pressure. This has to be tuned to keep dentries in memory as long as possible. Coupled with the fixed size of the dentry has table, the tuned amount of md_num_stripes, md_write_limit, md_sync_window and the actual number of files found, each system has it's own set of values before IO occurs to the disk. I once tried rebooting the system and expanding the dentry table to 2x its value and 3x it's value. I soon ran out of low memory on each boot. vfs_cache_pressure ------------------ Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

I think each system and usage is different. For some systems, you may want to disable cache_dirs, and others benefit from leaving it on. It all depends on how many disks, how many files and how much ram you have. Robj's point has merit. The only benefit of pausing cache_dirs on a parity check is if you see it affecting your parity check because it has to read the disks. Good tuning and/or disk selection may change that.