WeeboTech

i understand cache_dirs wants to keep the filelist? folder list? in ram.. seems like almost 4gb of ram for ~940gb of movies is a bit excessive. It's not how large the data set is, it's how many files. When the kernel tries to throw away the dentries, cache_dir is there forcing them to stay active. It's doing what it was intended to do. I don't have the code, but the chances are that it also adjusts the kernel to allow the dentries to be preferred to be in ram. I think it's an architecture limitation. It's an issue with low memory. When you run out of low memory, you start to get OOM errors. It's kernel tunings along with the md driver tunings. I'm hoping the x64 architecture will resolve this issue. I had 8GB and I had the same issues with a huge number of files. Another person had problems with an app and running out of memory. Here's the thread, you can also see what I put in my rsync script. http://lime-technology.com/forum/index.php?topic=28920.msg258077#msg258077 And another person who put the drop_cache line in the mover script. http://lime-technology.com/forum/index.php?topic=29798.msg267760#msg267760

It probably doesn't belong there. It belongs in the mover script and only belongs there if you are running cache_dirs AND/OR you have a huge number of files. I had issues without running cache_dirs. So did another member.

spin up your machine. drop your cache. See what happens. http://www.linuxinsight.com/proc_sys_vm_drop_caches.html

Normally I do not concern myself with cache_dirs. It has nothing to do with the code or anything like that. It's that I find it partially able to do what it's intended to do until you reach a limit of files. With 32 bit linux kernel there is only so much low memory you can use before you start having issues. This manifest itself in a number of ways, which sometimes look like a memory leak. I remember a while back a member was having trouble when running the mover script. In that we put some directives to drop the cache before and after the mover ran. It resolved his problem. If you are using a cache disk with mover, this may help. The down side is, all your disks will spin up once a day. It is a good time to do a collective file list for each disk. Then drop the cache, then let cache_dirs reload it and let the disks spin down. In my own case, I had so many small files on so many disks, I would get an out of memory error every night. cache_dirs aggravated that. Once I turned off cache_dirs and drop the cache after my nightly rsyncs, the problem went away. If I happen to execute two of these massive rsyncs in parallel, I would have out of memory errors again. I see that a DU is posted with disk sizes. It's not the size of the disk, but the count of the files. I had millions and millions of files. While many people here collect movies, I collect music. This has the result of many more files of a smaller size. hundreds of thousands of music files. I also collect source for review. Expand it, review it learn from it. leave it there. With my rsync_linked_backup solution, it worked really well, but since every dated directory had files hardlinked to the previous backup, it looked like may more files then actually existed. You can imagine the growing count of source code files that are constantly linked to prior directories every day. So it's more about the number of files and how many times you wade through it vs what other daily activities are caching date.

Look on newegg, they have a few models that have the chassis with a raid card. While you do not need raid, chances are you can use it in jbod mode, but noo guarantee with unRAID. you have to find out the chipset on the card. some of the Asmedia 6G chipsets work. I purchased this one. SANS DIGITAL TR4UTBPN 4Bay USB 3.0 / eSATA Hardware RAID 5 Tower RAID Enclosure (no eSATA card bundled) http://www.newegg.com/Product/Product.aspx?Item=N82E16816111149 Granted it has raid in it already, it supports usb and esata so it's probably more then what is needed. I purchased in the event it was repurposed. and I've repurposed it a few times to do different things. I use it with my HP Micro servers. Scour that thread for the controllers I used with it.

It's enough to handle unraid at a comfortable speed. I have 3, 2 are ESX servers, 1 is a N40L just as a file server. They run well.

I think this would be an interesting test.

I did not expect the numbers to have much effect on read, but I thought it would have 'some' effect on writes.

Save it where you get them from. i.e. disk.cfg especially if that's where the gui saves them.

You're getting carried away trying to make a tool that will do EVERYTHING plus make you some coffee. Focus on the core functionality, and let the users change their default settings if they like. Ever seen the movie Idiocracy? Remember the receptionist's computer in the hospital? That movie invented the Android OS years ahead of its time! Let the users make the change they want. Put in the documentation where to make the change in the gui or file. Later on after there are many tests and if you feel like revisiting, you can update the file automatically with some option switch. Or via some webgui presentation test. I would not do it at this point. Example. I want to run the benchmark without touching my own customized values. What I have works for very well for mel with no pauses and high burst write speed (Which is very important to me).

way better to use this method. eval $(egrep 'num_stripes|write_limit|sync_window' /boot/config/disk.cfg | tr -d '\r')

Try this one. fromdos < /boot/config/disk.cfg | egrep "md_num_stripes" or tr -d '\r' < /boot/config/disk.cfg | egrep "md_num_stripes"

I love this idea, if it were stored in the superblock.dat you could have a warning when it's older then a configurable time, or even automatically kick one off after a specifed number of days. if emhttp touched a file on the flash drive when a parity check/sync finished, it could be used as a timestamp semaphore also. Probably simpler then updating the superblock. Easy enough to test and get the date stamp of the file for displaying. Really keen on this idea too! +1 It looks like it's there. I can see it being used, but not saved. typedef struct mdp_superblock_v1_s { /* * Common information */ __u32 md_magic; /* 0 MD identifier */ __u32 major_version; /* 1 major version to which the set conforms */ __u32 minor_version; /* 2 minor version ... */ __u32 patch_version; /* 3 patchlevel version ... */ __u32 sb_csum; /* 4 checksum of the whole superblock */ __u32 ctime; /* 5 Array Creation time */ __u32 utime; /* 6 Superblock update time */ __u32 events; /* 7 Superblock update count */ __u32 md_minor; /* 8 preferred MD minor device number */ __u32 state; /* 9 array state */ __u32 num_disks; /* 10 number of disks in the array */ __u32 stime; /* 11 Last sync time */ __u32 sync_errs; /* 12 Last sync error count */ __u32 common_reserved[MD_SB_COMMON_WORDS - 13]; /* * Disks information */ mdp_disk_v1_t disks[MD_SB_DISKS]; /* * Reserved */ __u32 reserved[MD_SB_RESERVED_WORDS]; } mdp_super_v1_t; Looks to be exposed in /proc/mdcmd status output via status. Perhaps it's not being saved when updated.

My syslog is spotless so far. Are you asking me to double again, to 11776?!!! How many drives do you have? How many files do you have? If you are using cache_dirs, then it could hit some limit. Where I would see issues is using the locate package. Locate would go through the whole system tree and capture every file. Everything would be fine, until I ran that or did a massive rsync on my backup directory. I had 17 drives, but millions of files.

I love this idea, if it were stored in the superblock.dat you could have a warning when it's older then a configurable time, or even automatically kick one off after a specifed number of days. if emhttp touched a file on the flash drive when a parity check/sync finished, it could be used as a timestamp semaphore also. Probably simpler then updating the superblock. Easy enough to test and get the date stamp of the file for displaying.

I wonder if those stripes are using lowmem by any chance. If so, then that's asking for some serious trouble. Do you dare test with values twice that high? Tail the syslog from another console and watch if hell breaks loose. From what I remember, it uses low memory. A while back when I was testing this if I was to aggressive with my increase I would see the OOM issues crop up more frequently. The more drives you have, the more you are prone to an OOM condition when expanding these too aggressively. For my 4 drive HP MicroServer, I'm going to try and be as aggressive as possible for write speed purposes once the new script flushes out. I'm at 2048 now with good results, I didn't think about expanding it further. Color me interested!!!!

Please, add the md_num_stripes= with this line or at least start of line group. ) 100%: 49.7 MB/s @ md_write_limit=5568 uses 435MB I have an empty HP Micro server with unRAID under ESX that I can run tests on. I have a 4tb parity with a 4tb data drive and a 3tb data drive.

I can't comment on best bang for the buck, but I can say, I would choose the highest speed. @90%, Does using up double the amount of low memory make it worth the extra 6MB/s? For me, does using up the extra 500MB of memory make up for 10MB/s, Yes. Probably good only on the lower drive count systems. Until we are 64bit at least. Great job man!!!

At the current time, it will not, even with SSD as parity and data, it's still not what would be expected. However, with SATA III Speed and high quality SSD's, it could come pretty close.

This isn't what I said. (or meant). What I said was in response to... And my point is, people do writes locally so there is a benefit to maximizing md_write_limit to whatever gives you the fastest speed for your application, network or not. For most cases, it's network related for others it may not be. If I had the ability to to have higher write speeds by maximizing md_write_limit above network speeds, there is a benefit.. Especially if you have to do some full disk based operation. Even the chore of house keeping such as updating directory entries and/or superblocks would benefit from the highest speed possible. Remember when everyone saw this big speed boost in one of the RCs.. later on it turned out the superblock wasn't being updated properly. Regular housekeeping does take time and resources. There is a benefit to having the fastest write speed possible even if it exceeds your network. You would probably want to use lower md_write_limit settings if you want to preserve low memory.

Not all of us do these writes over a physical network. There are people virtualizaing with virtual 10GB virtual switches. There are people who do writes directly on the machine. i.e. downloads via torrent and/or news readers. People are using mysql for data warehousing. Many times I will move files via rsync from one area to another or decompress large archives on the array. I'm sure there are people who assemble files from newsgroups and would like it to occur as fast as possible. One of the things that I learned years ago when experimenting with unRAID and a P4 and XEON, When you benchmark a 1GB network from application to application without physical disk I/O or a filesystem you can achieve over 90MB/s. So the potential to move data at a very high speed does exist on the network. The potential to write data locally on the machine at a high speed exists with internal applications. Limiting writes to your network speed is probably only useful if an unRAID design has issues with low memory and the goal is to preserve that as much as possible.

Technical, anything is possible to be programmed. Practically, this may be splitting hairs. If you have it configurable to run a chosen drive that should be enough for the first go round. If you can iterate through arguments, then you can test each disk supplied. I.E. for i in $* do [ ! -d /mnt/disk${i} ] && continue TMPFILE=/mnt/disk${i}/${P}.test somefunction_to_test_the disk() ${TMPFILE} done this would allow people to test a range of disks with ./script.sh 1 5 7 4 3 This also means if they did not supply an argument for a disk write test, it would skip the code in the loop. I have diskrawread.sh you can look at via googlecode. It's different then a filesystem test. It's a read test to find the maximum read speed of a drive.

I have a script on my googlecode page called Identify drive. it's primitive, but it might have examples to teach you how to use the select command for a menu. https://code.google.com/p/unraid-weebotech/downloads/detail?name=identify_drive.sh&can=2&q=#makechanges

I would 'start' with b but only in that it selects /mnt/disk. Do not use of=/dev/sdg/testfile10gbzeros.txt as you will destroy the partition. use /mnt/disk1-24/filename to insure that you are writing to an existing filesystem. Put these at the start of your script and you can use your script for the basename of the temporary file. #!/bin/bash [ ${DEBUG:=0} -gt 0 ] && set -x -v P=${0##*/} # basename of program R=${0%%$P} # dirname of program P=${P%.*} # strip off after last . character TMPFILE=/mnt/disk${1} if [ ! -d ${TMPFILE} ] then echo "Supplied disk is not mounted. exit fi TMPFILE=${TMPFILE}/${P}.test then add trap "rm -f ${TMPFILE}" HUP INT QUIT TERM EXIT There's more then one way to skin a cat, we can use output of mount to find all mounted disks and present a menu. What the user would supply is the disk number. Later on you can calculate the other options or present a menu using select. (I'll send you some examples).

I would suggest reading /dev/zero as the kernel will provide 0's until you stop reading. There is also /dev/urandom but that takes more cpu cycles and I've found it to slow things down. Anyway, it's not out of thin air, it's from the kernel. It skips the need for a ramdisk. Personally when bench marking writes AND kernel tuning. I would not use the ram disk. I would drop the kernel caches. sync && echo 3 > /proc/sys/vm/drop_caches read from /dev/zero, writing to where I want to test with a specific limit. If you use the ramdisk approach, then you are holding ram that could possibly be used for buffering. For most people it would not matter, with my tuning and usage pattern it does. For example testing with sysctl vm.highmem_is_dirtyable=1 has a big effect with burstable buffering.