[cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up]

[To you "youngsters": Please allow us two old farts to reminisce; maybe it'll be of some interest to a few of you.]

 

Before I respond, allow me to put this in a chronological perspective (using a human lifecycle): I'd say that today, Unix is solidly in middle-age; it's healthy, accomplished, and has a long future ahead. The PWB/Unix that you referred to was first released in 1976, which I would characterize as Unix's early puberty; the time period you went hands-on, 1979-80, is getting close to its adolescence. When I first had hands-on with Unix and the kernel was Nov, 1974; that was the first release of the code from Western Electric (aka AT&T). Seven sets of tapes were mailed from Murray Hill; one came to me (on behalf of my employer). I would call that Unix's early weaning .

Linux ... feh!! ...

FYI, I first used, and did kernel development on, Unix before it even had dd (v4, 1973).

Were you one of those who used "adb -w" on the kernel while loaded in memory?    I fully respect anyone with that skill. 

adb? That's like solid food [cf: weaning]; there was no adb. There was only db, and truthfully, it is a misnomer to call db a debugger. Its limited functionality was to examine a (binary) executable (pre-execution), or to analyze a core-dump (post-execution). It could do nothing to help you in debugging a running program, or (obviously) a live kernel. [i will always remember my phone call to Ken Thompson, asking him where the (real) debugger was. When he told me there was none, I was honestly incredulous. I instinctively asked him "How did you and Dennis get all of this stuff working?" His answer: "We read each other's code."] And that is part of what identifies true software deities.

 

But, as a mere mortal, I still needed to debug my modified kernel. So, I hacked the header of DEC's ODT-11 (octal debugging tool for PDP-11) so that it was compatible with Unix ld (the [link-]loader). Adding this odt11.o to the rest of the kernel object files, ld gave me a kernel that I could debug, albeit painfully (no symbols--constantly referring to a nm listing for a symbol map). I believe I was the first person to actually debug (breakpoints, single-step, etc.) on Unix (kernel or user-space).

 

Not to evade your question , but by the time adb came around (1976-77), I was winding down my kernel hacking. [i guess I don't get your respect, huh? :)]

 

Oh my, time-traveling is tiring ... this geezer has to take a nap.

I'll follow-up the on-topic discussion of this (computer) memory exhaustion issue a little later.

 

--UhClem  "Forward! Into the past... back to the shadows, again..."

 

[cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up]

FYI, I first  used "dd" many many years before Linux was created.  It has not changed much over the years...

( It was on version 1.0 of CB-Unix.  (barely out of Bell-Labs))

Linux ... feh!! ...

FYI, I first used, and did kernel development on, Unix before it even had dd (v4, 1973).

 

The important point about dd, as it pertains to this discussion, is that the bs= and count= options have not changed. And, the (general) lesson to be learned by all (self included) is that it is important to read (and comprehend) the man pages for commands we hope to use correctly and constructively.

I knew it issued reads to the OS sized at the block size.  Never gave much thought to the "count" and its buffering prior to output. I'll have to look at linux source to see what it does these days. 

The description of the count= option has not changed in its entire 38+ year lifetime (except in a negligibly semantic sense):

May 1974: copy only n input records

August 2012: copy only N input blocks

The fact that it is a copy precludes any concern about buffering.

Regardless of what "dd" is doing,  the disk buffer cache will be keeping much of what it had recently accessed simple because it was most recently accessed.

It isn't really keeping it (in an active sense); it has just not yet overwritten it with anything else. Regardless, this can not be the source (nor excuse/explanation) for any shortage of user-space memory. [Yes, a perverse, and privileged, user can "manufacture" a problem by setting excessively agressive memory tuning parameters. If so ... "you make your bed, you have to lie in it."]

In the same way, cache_dirs is just using the "find" command, and it will force a lot of the disk buffer cache to be involved if you have a deep/extensive directory hierarchy.  Between them,  you can run out of "low" memory.

No, I don't believe it really is "in the same way". In the dd example, only the buffer cache is in play, and in a very simple/straightforward manner. In the case of your find/cache_dirs example, there is likely some "race condition" provoked by an interplay of the buffer cache, the inode cache, and the directory-entry cache. If you can really cause an error condition this way, then it is a system bug (technically). [but nobody is both willing and able to fix it. (You know, like the national budget problem )]

 

--UhClem  "(readme) (doctor memory)"

 

[cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up]

I fully understand that "cylinders" have not been used for 20 years (probably much more)

The issue I was faced when originally writing the preclear script was in selecting an appropriate "block size" when reading and writing to the disks.  I used the output of the "fdisk" command as my guide.  I figured the disk geometry would probably report a size it could handle.

No. (In the chronological context of this forum) Totally forget about (disregard!) DISK GEOMETRY.

"fdisk" presented a line like this (from a sample IDE disk on one of my servers):

Units = cylinders of 16065 * 512 = 8225280 bytes

The preclear script then read, by default, 200 "units" of data at a time with a "dd" command looking something like this (for that disk)

dd if=/dev/sdX bs=8225280 count=200 .........

 

The amount of memory used for a single read request was then 8225280 * 200 = 1,645,056,000 bytes.

No. dd read 8225280 bytes at a time. (the entire dd run consisted of 200 such reads). Each read() (re-)used the same ~8MB (user-space) buffer.

Now, with larger 3TB disks, and a much larger different "Unit" you can easily run out of memory.

I don't have any >2TB drives, so I don't know how fdisk -l reports them, but if you do get a much larger "Unit", you have no grounds for complaint. Instead, consider yourself fortunate that you didn't get bit earlier (for following the folly of disk geometry).

I agree, there needs to be a limit, but a multiple of 4k makes no practical difference at all when you are asking for 8225280 bytes or more at a time.

Practical difference or not, it is just good practice to use the same basic unit (and multiples thereof) as the OS. (it's a corollary of the Law of least surprises )

 

[cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up]

basically, you ran out of free memory.

 

Apparently, on the 3TB drives you will need to use the -r, -w, and -b options to limit the memory used by the preclear process.

This is because those parameters were originally sized when 1 TB drives were common.  At that time the preclear script was designed to read and write a ==>cylinder<== at a time.

Cylinder?  As a design criterion (such as you allude), the cylinder has been obsolete for ~20 years, even moreso in the last 5-8 years. Just change your default to something that "feels" right. And, totally forget about "disk geometry"--all that does for you is create a chunk size that is NOT a multiple of 4K.

 

--UhClem "The times they are a'changing."

 

[Script to enable SATA Controllers without kernel modification]

... I've a rocketraid 2720sgl pci-e 2.0 that is attaches 4 WD 3TB red drives via mini-sas to sata.  I can't see them when I boot my server up.

You need to add the proper identifier that matches your controller to the script. Do this by amending the PCI_IDS= string to include your new identifier.

 

Your identifier should be one of:

 

1. 1b4b9485

2. 11ab9485

3. 11032720

 

Try them, one at a time (separate trials), in the order of preference above. The first one that results in you "seeing" your drives will (probably) work best [in the unRAID scenario].

 

Good luck.