Howdy!

 

Here's my setup:

 

Version: 4.2beta4.

Drives:

 

parity device:  pci-0000:00:1f.2-scsi-0:0:0:0 (sda) scsi-SATA_WDC_WD10EACS-32_WD-WCASJ0098525

disk1 device: pci-0000:04:03.0-scsi-3:0:0:0 (sde) scsi-SATA_WDC_WD7500AAKS-_WD-WCAPT0146702

disk2 device: pci-0000:04:04.0-scsi-1:0:0:0 (sdg) scsi-SATA_SAMSUNG_HD400LJS0H2J1KLA00327

disk3 device: pci-0000:04:04.0-scsi-3:0:0:0 (sdi) scsi-SATA_SAMSUNG_HD400LJS0H2J1KLA00325

disk4 device: pci-0000:04:04.0-scsi-2:0:0:0 (sdh) scsi-SATA_SAMSUNG_HD400LJS0H2J1KLA00326

disk5 device: pci-0000:04:04.0-scsi-0:0:0:0 (sdf) scsi-SATA_SAMSUNG_HD400LJS0H2J13LA00143

disk6 device: pci-0000:04:03.0-scsi-2:0:0:0 (sdd) scsi-SATA_MAXTOR_STM35006_9QG07NDC

disk7 device: pci-0000:04:03.0-scsi-1:0:0:0 (sdc) scsi-SATA_WDC_WD10EACS-32_WD-WCASJ0178650

 

Mobo: Asus p5b-vm do.

Controllers: Promise Sata TX4 (4 ports each), 2 cards.

 

All of the drives minus the parity drive are plugged into the PROMISE cards. The Parity drive is plugged into one of the Intel ports.

 

Network is the P5B-VM DO's onboard NIC, the Intel 82566DM chipset one. Gigabit connection, via an, albeit cheap, gigabit switch. All of the client machines can talk to each other at gigabit speeds, no issue.

 

The problem I have, though, is this:

 

Transferring files to the server, via user share, or direct-to-drive, the network usage/transfer speed spikes high, 20-30-40% utilization on the gigabit NIC; this is good. Then, though, it drops to 0. Basically, it seems to transfer into a "cache" on the server, then, if viewing/refreshing the web interface, the data is written to the parity/destination drive, and then the network utilization kicks up again. It keeps doing this until the file transfer is done.

 

Now, is this normal? Should this setup be capable of sustaining gigabit speeds? If not, it's fine, it's not an issue. It transfers fast enough for my needs, I'm just wondering where the proverbial "bottleneck" is in this situation. The parity drive is slower; it's one of the WD 1TB drives, and it is slower than the rest, I won't lie... but slow enough to make this happen? That I don't know.

 

I suspect the PCI bus as a bottleneck as well, since, well, it's obviously limited to 133MB/s, shared between, atm, 7 drives.

 

Thanks for any opinions!

 

Edit: FWIW, transferring files from the unraid server to a client machine has no drop offs in speed; it's steady until the end.

 

Edit: Nevermind, searching showed me... well, everything! I apologize for the post

There are a million posts on network speed, but what you are seeing is how unRAID works. I will leave the technical explanation to an expert here, but in short; it's working as it should.

Searching is far too difficult. Fair enough; Searching did show my answer. My bad

I hope you don't mind this very off-topic post, but you started a very exciting (for me anyway) stream of thought.

 

First a comment on the PCI bottleneck concern, your smart idea of putting the parity drive on the PCI Express-or-better bus, and off the PCI bus ensures that you are not bottlenecked, in my opinion.  I don't believe the PCI bus is a problem for single disk access, but simultaneous device access across the PCI bus creates traffic contention, whether it's 2 drives being read or written, or one drive and a PCI gigabit network card.  In the unRAID system, reading from a drive only involves that single drive, 1 read I/O per block.  Writing to an unRAID drive involves 2 drives, the data drive and the parity drive, with 2 read I/O's and 2 write I/O's.  So keeping the parity drive off the PCI bus should probably be axiomatic for better performance.

 

Now I digress...  I started thinking about what actually happens during an unRAID write, and why it is so much slower, and causes the stream stoppages mentioned above.  I'm a programmer and there is almost nothing more exciting to me than optimization, especially of a critically important algorithm.

 

Here's what I think is happening:  (some speculation from here on)

  we receive a data block to write

  read the current data block from the corresponding position

  read the current parity block from the corresponding position

  do the parity processing/calculations

  write the new data block

  write the changed parity block

 

The part that is most interesting to me is the parity processing, but unfortunately this is a very I/O-bound routine, so if the parity code is even halfway decent, nothing you improve there will matter compared to improvements in the I/O parts.  The only I/O improvement I can think of, is to make sure the 2 reads are simultaneous and the 2 writes are simultaneous - no reading, waiting for return code, then do the second read, and the same with the writes.  You have to handle error returns, but they are the exception.  I have no idea if this is what Tom has done, and am in no way commenting on his code, just imagining how I would do it.  The data read, followed by the parity read, followed by parity processing, followed by the data write, followed by the parity write, would take about twice as long as doing the reads, then the parity work, then the writes.  I have no experience with the Linux file system, but there must be a way to do nonblocking reads and writes, or if not, then opening a second thread to initiate the simultaneous I/O.

 

The parity processing is what excites me the most, because it seems to cry out for a good assembly routine, of bit block manipulation.  Again, I have no idea what Tom has done, and he may well have something better than my ideas, but here's what I would do.

 

  we have a block of data to write, stored in BufferNew

  we have read the old data block, into BufferOld

  we have read the parity block, into BufferParity

  we NOT BufferOld  ;; flips every bit

  we XOR BufferNew into BufferOld

      ;; this gives us a perfect bit pattern in BufferOld: the 0 and 1 bits tell us exactly which of the parity bits to keep or flip

  ;; at this point we continue a set of block bit operations, ANDs, ORs, NOTs, and/or XORs, using BufferOld on BufferParity, but I haven't worked out the right sequence yet

  ;; this results in an updated parity block in BufferParity which we can write to disk

 

Using block bit manipulation in assembler would result in a routine that could well be 100 times faster than even the best compiler-optimized code of a programmatic loop of the change analysis and parity bit changes.

 

For any that have actually read this far, I apologize, especially if you feel I have wasted your time.  This kind of analysis and design I find especially fascinating.

 

Rob, your contributions to this board give you the right to make no apologies when you post, regardless of how off-topic it may be.

 

I hope that Tom looks at this and comments, whether it be a "nice job" or a subtle chuckle. 

 

 

Bill

 

The parity processing is what excites me the most, because it seems to cry out for a good assembly routine, of bit block manipulation.

 

Rob,

 

Your analysis is about as good as any.  Nice to see a fellow software developer in here. 

 

I've not looked into the specific algorithm involved in writing to the /dev/md device... however...

the source code involved is on our unRaid servers.

 

log in via telnet and "cd" to /usr/src/linux/drivers/md

 

The source code for xor.c has the following line within it:

#include <asm/xor.h>

 

Looks to me as if the "md" device already uses optimized assembler routines.

 

The sequence of reading the parity drive block and data drive block (hopefully in parallel), XOR'ing their contents, then writing both devices (again in parallel) is where any possible improvements in performance will occur.  Odds are good that the XOR assembly has been refined over time as part of the Linux RAID "md" device and that it is not the bottleneck.    There might be a way to improve write performance if either the reading of the existing data, or writing of new are occurring serially rather than in parallel.

 

Now, we'll have to download a full copy of kernel source to see the actual assembly routine.  Apparently, since Tom did not modify asm/xor.h, he did not include it as part of his source distribution.

 

Joe L.

First a comment on the PCI bottleneck concern, your smart idea of putting the parity drive on the PCI Express-or-better bus, and off the PCI bus ensures that you are not bottlenecked, in my opinion.  I don't believe the PCI bus is a problem for single disk access, but simultaneous device access across the PCI bus creates traffic contention, whether it's 2 drives being read or written, or one drive and a PCI gigabit network card.  In the unRAID system, reading from a drive only involves that single drive, 1 read I/O per block.  Writing to an unRAID drive involves 2 drives, the data drive and the parity drive, with 2 read I/O's and 2 write I/O's.  So keeping the parity drive off the PCI bus should probably be axiomatic for better performance.

 

I mostly put the Parity onto the motherboard SATA port since, when arranging the drives, it just made more logical sense for me at the time. I figured I'd fill the PCI cards up first, seeing as how they were the "weaker link" in my mind, so I kept the Parity drive on the mobo. As you mentioned, usually only one drive is read at a time, since the data isn't striped, but during a write operation, the PCI bus might become a point of contention. Whether or not it'd actually make a difference or not is debatable, hehe.

 

I don't know enough about the programming logic to make comments about the rest; I was honestly hoping it was a bottleneck on MY end; it'd give me a reason to buy a shiny new Hitachi or Seagate 1TB drive to replace my "slow" TB parity drive .

Seems like there's a few threads on this with at least one contention being that the drives are simply going as fast as they can and are the bottleneck. I honestly don't know but am as frustrated as everyone else

 

A data point... My IDE system now has ONE SATA drive in it run from the M/B - that is in the Parity slot. I see no increase in either transfer speeds or Parity checking on that system. My other system is 4 SATA and 2 IDE drives - it parity checks at over double the speed of my first system but admittedly it isn't fully loaded with drives. My mostly IDE system is also on older software but I do not think that is making a difference in this case.

 

As I understand it the XOR operationis the chokepoint in software RAID and in many hardware RAID systems. I'd agree that it is probably highly optimized. It might be interesting to test to see if it performs better on say an AMD CPU vs Intel or if using something crazy like a Vid card to do the calc might speed things. I do not know if the XOR is our chokepoint and I think it would be VERY interesting if someone did some performance tracing on the code to try and figure this out or at least more testing. If it's really the disks, and the argument that it is seems strong to me, then I guess it is what it is for now. That adding additional memory cache speeds transfers though seems to point to at least some room for improvement and I cannot help but wonder if network stack tuning might help too. Someone is trying to use a TIVX box with unRAID and is finding that it cannot keep up, that is troubling especially if it turns out that a Windows server could. It would be VERY interesting to see how unRAID would perform with some SSD - that would certainly expose software weaknesses if the newer faster SSD were used! It's only a matter of time before those drop in price to the point where this coud be tried IMO.

 

Anyway, yeah this is something that I think frustrates many of us right now! <shrug> I'm moving hundreds of Gigs of DVD images around and this is rearing it's head constantly. No timeouts, no dropped connections, just time consuming.

very time consuming i get spikes close to 30-60% network, then to 0% as well...and it's a pain if your unRaring something to the unraid array, b/c the unrar freezes also, which seems to freeze my system as well for a lil bit every 5-10 seconds or so, and i have a powerful system....

when i first installed unraid and wanted to copy data over to the drives (mounted ntfs drives), i didn't even assign the parity drive, copied everything over at full speed (60MB+/sec) and then assigned the parity drive and ran the sync...otherwise it was taking too long

 

but anyway, i haven't tried, but what happens if you try to copy 2 files to two different drives at the same time?

 

would it be possible, to maybe just copy everything over w/o running the parity in parallel? like...set a pointer to where the start of the data (mainly for large copies), pointer to the end of file, and then run the parity on the server AFTER the file has been fully transfered? ... or something like that

 

also, is linux able to use the power of dual core cpus?

 

 

would it be possible, to maybe just copy everything over w/o running the parity in parallel? like...set a pointer to where the start of the data (mainly for large copies), pointer to the end of file, and then run the parity on the server AFTER the file has been fully transfered? ... or something like that

 

 

 

Yes, you can do that but you will find the ROI isn't there except for the largest of copies.  Save ten minutes then wait six hours for parity since the entire array gets recalced.  If you were doing a mass transfer from, say, a full 500GB PC drive to Unraid, then it *may* make sense.  Actually, I just did a "back of the envelope" calc and showed 450GB would be the break-even assuming you gained a 25MB/sec speed advantage (a swag) and assuming parity takes six hours (another swag).  Someone can double-check my math ....

 

Assuming I did the math right, that means your first load of data could be done this way, but after that just go ahead and deal with the slower speed.

 

 

Bill

thats not what i was saying, i didn't say to run full parity after the transfer, that would be a waste of time

 

what i'm saying is, if when transferring large amounts of data, you can set a pointer as to where on the hdd the new data starts, and another point to where on the hdd the new data ends, if thats possible, then run the parity update on just that part of the HDD after the file has been transferred

 

basicly, unraid could control what area of the hdd its written to and have pointers for start/end positions of the data and update parity only on the area it was written to after data transfer is complete...i dunno if that makes sense or not

 

basically if disk1 had 100gb of free data and you were transferring 40gb worth of data, u can set start/end pointers for each free section of the hdd that the data was written to, and then just run the parity check on those areas after transfer was complete, so transfer wouldn't suffer and parity would complete faster as well...

How would you set the pointers?  More specifically, how would the typical user set the pointers?

 

Perhaps the system tracks changes, so you would do some sort of "suspend parity/track changes, then recalc based on changes" in the background so the user wouldn't have to do anything other than those two steps?

 

Interesting concept.

 

 

Bill

the typical user would have to do nothing, everything should be taken care of by the OS (unraid/linux)

 

i don't know the the file system works, or how fragmentation will affect it and how it's decided where each bit/byte of data goes, but here's the basic concept

 

lets take disk1, which is a 500gb disk, and has 100gb of free space, and lets say the disk is a little fragmented, so the 500gb of data looks like this

 

100gb used | 20gb free | 80gb used | 40gb free | 200gb used | 40gb free | 20gb used

 

lets say your copying a 40gb file --- unraid should know where exactly on the hdd its copying it, what sector etc, and record start/stop position for that block and lock those sectors for that file being copied so no other files can be copied there if more than 1 person is copying at the same time...

and lets say unraid decided to copy 10gb in the first block of free space, 20 in the 2nd block of free space, and 10gb to the 3rd block of free space, there would be a start/end pointer set on each of the sections the data was copied to and then parity would be run only on those sections.

 

now this is vague, but hopefully u get the idea, i don't know if its possible or not, but sounds like it should

OK, it's getting more interesting.

 

It sounds like your idea could be more succinctly stated as, "perform all parity calculations only when no writes are occuring and only calculate them where it needs to, not just 'calc everything'".

 

I like it.  The downside is that a user could continuously write to the system, creating a situation in which parity is never calculated.  Perhaps a user-definable toggle not dissimilar to drive spin-down time.  "Parity wait".  This could be 10 minutes (i.e. start parity ten minutes after the last write), 20, 30, ... or even zero (the value used today, calc them on the fly).  It could also be simpler as a "wait: yes/no" where it waits XX minutes if yes is selected (number doesn't really matter, imo).

 

BTW, I know my response doesn't include much of your previous post.  Suffice it to say that the system tracking changes would be a critical component necessary to make this idea work.

 

You should this to the "ye old laundry list".

 

 

Bill

Delayed parity calculation strikes me as a bad idea. If another drive experiences problems during one, that drive will be unrecoverable unless it could be guaranteed that any empty sectors always contribute only, say, 0s to the parity calculation. This would make file deleting as expensive as file writing, would put lots of additional demands on the parity drive, and could open up for any manner of race conditions that not only can leave the parity inconsistent but might also be extremely hard to detect. The bottom line, I think, is that writes must be atomic.

Delayed parity calculation strikes me as a bad idea. If another drive experiences problems during one, that drive will be unrecoverable unless it could be guaranteed that any empty sectors always contribute only, say, 0s to the parity calculation. This would make file deleting as expensive as file writing, would put lots of additional demands on the parity drive, and could open up for any manner of race conditions that not only can leave the parity inconsistent but might also be extremely hard to detect. The bottom line, I think, is that writes must be atomic.

 

I agree that this must be more thoroughly thought out.  However, even the current functionality has risks when, for example, a drive fails during a full parity check or during a rebuild.  Perhaps the risk is not worth the added performance, but we won't know unless someone does a proper assessment.

 

 

Bill

Delayed parity calculation strikes me as a bad idea. If another drive experiences problems during one, that drive will be unrecoverable unless it could be guaranteed that any empty sectors always contribute only, say, 0s to the parity calculation. This would make file deleting as expensive as file writing, would put lots of additional demands on the parity drive, and could open up for any manner of race conditions that not only can leave the parity inconsistent but might also be extremely hard to detect. The bottom line, I think, is that writes must be atomic.

 

File deletion does not need to have delayed parity, as deleting a 40gb file takes a few seconds.

How would a failed drive during a delayed parity write cause the drive to be unrecoverable?

If a file is being copied to disk1 and the copy is successful, the delayed parity now starts after the transfer to update itself based ONLY on the changes to disk1 (so only the parity and disk1 are being accessed now), if disk2 fails, it would still be recoverable, as long as the parity update finishes from disk1.

 

also, just wondering, what happens/or how is it handled, when a parity build or a parity check or a file copy is taking place, and someone decides to copy something else at the same time to either the same drive or another drive....would the parity still be in sync?

How would a failed drive during a delayed parity write cause the drive to be unrecoverable?

 

Consider an array with two data disks, disk1 and disk2, and a parity drive, parity. Imagine that, by chance, disk1 contains all 0s and disk2 contain all 1s. Parity will contain all 1s. Now write 1s everywhere on disk1 without updating parity. If disk2 fails at that point, the only guess we can make about its content AT THAT POINT is that it's all 0s.

 

If a file is being copied to disk1 and the copy is successful, the delayed parity now starts after the transfer to update itself based ONLY on the changes to disk1 (so only the parity and disk1 are being accessed now), if disk2 fails, it would still be recoverable, as long as the parity update finishes from disk1.

 

While technically correct, this does not qualify as a solution in my book. We agree that the bottom line is that writes must be atomic and the question we are faced with is how long we are ready to have a system monopolised by one operation. Fractions of a second are OK with me, but anything longer than that does not make for a file system as far as I'm concerned.

 

File deletion does not need to have delayed parity, as deleting a 40gb file takes a few seconds.

 

I alluded to the possibility of having non-atomic writes. To accomplish that, we could enforce that empty sectors are all 0s and that the parity drive reflects that. In the case of failure ahead of a delayed parity computation, we can reconstruct everything on the spot by simply using 0 instead of the non-committed writes. The extra stress I referred to is the overhead of ensuring all 0s in empty sectors.

 

 

Consider an array with two data disks, disk1 and disk2, and a parity drive, parity. Imagine that, by chance, disk1 contains all 0s and disk2 contain all 1s. Parity will contain all 1s. Now write 1s everywhere on disk1 without updating parity. If disk2 fails at that point, the only guess we can make about its content AT THAT POINT is that it's all 0s.

 

like i said, you would not lose a disk that way, you would just need to update parity based on changes to disk1, the parity update reflecting disk1 would have to finish b4 you can rebuild your failed disk2.

 

We agree that the bottom line is that writes must be atomic and the question we are faced with is how long we are ready to have a system monopolised by one operation. Fractions of a second are OK with me, but anything longer than that does not make for a file system as far as I'm concerned.

 

i have no idea what your talking about here...monopolized by what operation? the way it currently works correct?

well, right now network transfers its very slow/delayed, i think having an option for the user to enable this would be useful.  I want a constant transfer speed of 60MB+/sec as i used to get when running windows2003 server, not spikes after every 5 seconds.  I rip my HD-DVDs to my server, so my data transfers are usually 20gb+ in size,and this is just painstakingly slow.

All of us who are running server know a thing or two about how things work and importance of having the parity in sync...even allowing this as a turn on/turn off button would be helpful when copying large files, and then the server can build the parity info on the changed parts after the transfer.

but yes the ideal situation would be synchronously writing and updating parity w/o the slowdown in transfer....

 

I alluded to the possibility of having non-atomic writes. To accomplish that, we could enforce that empty sectors are all 0s and that the parity drive reflects that. In the case of failure ahead of a delayed parity computation, we can reconstruct everything on the spot by simply using 0 instead of the non-committed writes. The extra stress I referred to is the overhead of ensuring all 0s in empty sectors.

 

according to you, for the file being copied, each sector at the same location for each drive must be 0 for it to work, which would be pretty hard to accomplish as disks are different sizes w/ different amounts of data.

 

We agree that the bottom line is that writes must be atomic and the question we are faced with is how long we are ready to have a system monopolised by one operation. Fractions of a second are OK with me, but anything longer than that does not make for a file system as far as I'm concerned.

monopolized by what operation?

 

The write.

 

I alluded to the possibility of having non-atomic writes. To accomplish that, we could enforce that empty sectors are all 0s and that the parity drive reflects that. In the case of failure ahead of a delayed parity computation, we can reconstruct everything on the spot by simply using 0 instead of the non-committed writes. The extra stress I referred to is the overhead of ensuring all 0s in empty sectors.

according to you, for the file being copied, each sector at the same location for each drive must be 0 for it to work,

 

No.

very time consuming i get spikes close to 30-60% network, then to 0% as well...and it's a pain if your unRaring something to the unraid array, b/c the unrar freezes also, which seems to freeze my system as well for a lil bit every 5-10 seconds or so, and i have a powerful system....

when i first installed unraid and wanted to copy data over to the drives (mounted ntfs drives), i didn't even assign the parity drive, copied everything over at full speed (60MB+/sec) and then assigned the parity drive and ran the sync...otherwise it was taking too long

 

but anyway, i haven't tried, but what happens if you try to copy 2 files to two different drives at the same time?

 

would it be possible, to maybe just copy everything over w/o running the parity in parallel? like...set a pointer to where the start of the data (mainly for large copies), pointer to the end of file, and then run the parity on the server AFTER the file has been fully transfered? ... or something like that

 

also, is linux able to use the power of dual core cpus?

 

 

Check out this link to an old post on a linux samba mailing list http://lists.samba.org/archive/samba-ntdom/2001-March/018285.html

 

It describes the exact network symptoms as unRaid is experiencing.    In its case, when the network freeze occurred a program "kupdate" was using all the CPU time.  That is the process that wrote the data to the physical disks from the disk cache.

 

Now, this was in 2001... but the symptoms seem to be the same.

 

Since then, the "kupdate" process was renamed to be "kupdated"  It is described here: http://www.cs.cmu.edu/~mukesh/hacks/spindown/t1.html

 

In kernel 2.6, pdflush replaces bdflush and kupdated from 2.4 and is responsible for periodically flushing dirty pages in order to ensure that there is enough free memory and to prevent users from losing data when their computer crashes and things are still in the buffer cache instead of on disk.  It is described here http://www.westnet.com/~gsmith/content/linux-pdflush.htm

 

It almost seems as if pdflush has a higher priority than the network IO and when it is active IO activity is blocked.  Perhaps some tunning here is in order.  It should be writing the "dirty" disk blocks in memory to the physical disk in the background whenever it gets an opportunity, and not stopping everything else while it is doing it.

 

Someone on the web described a patch to "renice" the priority of pdflush to keep it from blocking other activity.  It was against an older kernel...

 

We could test if this is the cause of the network IO start/stop spikes by

typing :

renice 0 -p `pidof pdflush`

154: old priority 0, new priority 0

153: old priority 0, new priority 0

Looks like it was already at priority 0, perhaps a priority of 1 or more might be better so it is at a lower priority than normal IO from the  LAN.  Some experimentation is in order.

 

Perhaps something like this: (lower numbers = higher priority)

renice 1 -p `pidof pdflush`

154: old priority 0, new priority 1

153: old priority 0, new priority 1

 

Joe L.

 

 

 

 

 

tried what you listed above:

 

root@Tower:~# renice 0 -p `pidof pdflush`

4416: old priority 0, new priority 0

3569: old priority 0, new priority 0

 

here's the net graph with 0:

 

root@Tower:~# renice 1 -p `pidof pdflush`

4416: old priority 0, new priority 1

3569: old priority 0, new priority 1

 

here's the net graph with 1:

 

you see the spikes i highlighed in the picture above? It looked like it went back to priority 0, so i typed in priority 1 again and this is what i got for the first spike:

root@Tower:~# renice 1 -p `pidof pdflush`

4483: old priority 0, new priority 1

4473: old priority 1, new priority 1

 

for 2nd spike:

root@Tower:~# renice 1 -p `pidof pdflush`

4493: old priority 0, new priority 1

4491: old priority 0, new priority 1

 

as you can see, the numbers have changed and after typing that in again, it stopped spiking.

 

I dunno what all this means, but thought i'd test it out for u.

 

I dunno what all this means, but thought I'd test it out for u.

From your experiment I can see several things...

 

1. the pdflush processes are being re-generated.  Apparently, our server is set up to have two of them running at a time, when one wakes up and writes the disk buffer cache to disk it then exits and a new pdflush instance is created.  You can see this since every time you did a "renice" is was against different process IDs. (the number before the colon in the renice output message)

 

This explains why the nice value of "0" is restored after a period of time. It is the normal value for the new process once the old pdflush exits after writing its set of blocks to the physical disks.

 

2. When the nice value is set to a "1" the pdflush process has 80% (from what I've read) of the priority of processes running at nice=0.  This value seems to stop the huge spikes you are seeing in transfer speed.  It seems that things smooth out a bit.

 

3. Even with a nice value of 1, the initial transfer speed seems to spike high once. (perhaps until some buffer somewhere is filled)

 

4. Looking at your spiky screen capture, for each "spike", the initial spike high of 50 to 75% bandwidth seems to be followed by a plateau of network activity at around 10 to 15% of network bandwidth.  This is then followed by a period of near 0% network bandwidth, then followed by the next cycle's high spike, where again between 50 and 75% of bandwidth is used. (again till some buffer fills)  This cycle then repeats... an initial high spike, then activity near 10-15% of bandwidth, then no activity while the disk buffers are written to the physical disks by pdflush.

 

5. With the nice value set at "1" the average network usage seems to be in the 10 to 15% range.

 

Now... we don't know if it is the "md" driver that is sleeping while "pdflush" is active with a nice of "0", or if it is the samba process, or the network card driver.  It could be any one of them.

 

If it is the samba process, we might be able to see the spikes go away if we made it a higher priority (instead of making pdflush a lower priority)

renice -1 -p `pidof smbd`

10936: old priority 0, new priority -1

10935: old priority 0, new priority -1

9523: old priority 0, new priority -1

If the spikes remain when smbd is set to -1, then my guess the issue is with either the network card driver or the "md" (unRaid multi-device) driver.  I will warn you... setting smbd to -1 will give it priority over most other processes, so you might not be able to type in the telnet window till it is idle.  (I tried this... I was able to type to reset smbd to a nice of "0)

 

If it is not improved when smbd is set to a -1, then my prediction is that ftp or nfs file transfers to unraid will suffer the same blocky/spiky performance as samba since "pdflush" is involved.

 

So... I "donno" what this all means either.  Clearly, more experimenting is in order.  The average transfer rate of 10-15% may be hitting some other bottleneck.  The overall transfer speed of the spiky network flow may not be much different than when it is not spiky. It may still average out to 10-15%. Timing tests are in order to learn if overall speed is better or not when not spiky.

 

There are tunable parameters we can set for pdflush in /proc/sys/vm .  perhaps it can be set to block less frequently.

 

Perhaps Tom will have some thoughts when he reads this thread.

 

Joe L.

I found another manifestation of this problem.  In converting my server from Windoze and NTFS drives to unRAID, I was mounting the NTFS partitions, and copying disk-to-disk on the server.  If at the same time I tried to copy files to the server via Samba over the lan, it the lan copy choked off to almost nothing, and often timed out, even though the disk-to-disk copy involved two different disks (such as disk 6 and 7) while the lan copy was to a third (disk 1) and the CPU utilization by the copy was only 20%.

 

As soon as the disk-to-disk copy completed, the lan copy picked back up to regular speed.

 

Using renice on pdflush did not help.