Re: How much to malloc(), without running into swap...?

["Benjamin C.R. LaHaise" <blah@kvack.org>]

On Sat, 6 May 2000, Andrew Clausen wrote:

> Hi all, (please cc me)
> 
> I'm hacking GNU Parted, which can (amongst other things) resize
> file systems.  It's performance is GREATLY improved if large
> disk buffers are used, provided it doesn't need to swap to access
> the buffers ;-)

mlock() is your friend.

> So, how can I maximize the buffer sizes, without running into
> swap?  Note: I don't want to disable swap, because a certain
> (large) minimum is required for storing metadata, etc., so
> low-memory machines might want to use a swap (despite it being
> slow).
> 
> So, I want to know:
> (a) how much I can malloc() without swapping
> (b) how much I can malloc() with swapping
> 
> Also, I presume all IO is going to have to go through the buffer
> cache, etc., so having a larger buffers means more consumption
> on the kernel side of things.  OTOH, it can probably kick out
> old cached data fairly quickly (i.e. data used by other programs)
> Any ideas on how to do the calc's?
> 
> BTW, is /proc/meminfo a good idea?

That's probably your best bet if your app is the only thing running in the
system.  Note that you'll get some feedback about things at mlock time --
if there just isn't enough memory free, mlock() will fail, so you can use
that to scale back your memory requirements if other parts of the system
are eating memory.  With swapping, things get trickier -- you'll probably
want to minimise the mlock()'d data.  It would probably be better to avoid
swapping altogether for your app since it would cause a ridiculous growth
in the running time (and probably can be avoided).

You might want to look into using the raw devices when possible since you
can probably make a better guess at the best way to order your io than the
buffer cache currently will.  Doing that will also get the added benefit
of avoiding any excess swapping that might be generated by competition for
memory by the buffer cache.  Hope this helps,

		-ben

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/