Went back and tried a few things.

Finally figured out that the problem can be reproduced with a 
simple shared memory segment loader utility we have.  No 
threads, no forks, nothing fancy. Just create a segment and read 
the contents of a big file into it.  Two segments actually.  The 
only difference is the accessing program has to be run three 
times instead of two times to produce the failure.  You might be 
able to accomplish the same result just using 'memset()' to 
touch all the memory.

Then tried this out with the F9 kernel 2.6.26.5-45.fc9.x86_64 
and everything worked perfectly.

This is all I can do.  Have burned way to many hours on it and 
am now retreating to the warm safety of the RHEL kernel.  Only 
reason I was playing with the kernel.org kernel is we're trying 
to get an Intel 82575 working with the 'igb' driver in 
multiple-RX-queue mode and the 'e1000-devel' guys said to use 
the latest.  However that's looking like a total bust, so it's 
time to retreat, wait for six months and hope it's all working
by then with a supported kernel.

I've attached the 'strace' files.  Don't know where those 
'mmap's are coming from except that perhaps in a library 
somewhere.  There are none in our code.

Good luck.