Re: maximum memory limit

Re: maximum memory limit

[Rik van Riel]

On Tue, 8 Feb 2000, Lee Chin wrote:

> Sorry if this is the wrong list, but what is the maximum virtual
> memory an application can malloc in the latest kernel?
> 
> Just doing a (for example) "malloc(1024)" in a loop will max out
> close to 1GB even though I have 4 GB ram on my system.

The kernel supports up to 3GB of address space per process.
The first 900MB can be allocated by brk() and the rest can
be allocated by mmap().

Problem is that libc malloc() appears to use brk() only, so
it is limited to 900MB. You can fix that by doing the brk()
and malloc() yourself, but I think that in the long run the
glibc people may want to change their malloc implementation
so that it automatically supports the full 3GB...

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Matthew Kirkwood]

On Tue, 8 Feb 2000, Rik van Riel wrote:

> Problem is that libc malloc() appears to use brk() only, so
> it is limited to 900MB. You can fix that by doing the brk()
> and malloc() yourself, but I think that in the long run the
> glibc people may want to change their malloc implementation
> so that it automatically supports the full 3GB...

The glibc manual says that for allocations much greater
than the page size (no, it doesn't quantify "much") it
will use anonymous mmap of /dev/zero.

It's probably a bad idea to allocate over a gigabyte in
1K chunks anyway...

Matthew.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Mark Hahn]

> Problem is that libc malloc() appears to use brk() only, so

modern libc's certainly use mmap for large mallocs.  but this can be a 
serious problem: I corresponded with someone who had a binary app that 
did many small mallocs, and he was pissed that his 4G box could only malloc
900M or so.  this happened because __PAGE_OFFSET and TASK_SIZE were 3G, 
but TASK_UNMAPPED_BASE, where mmap's start, is TASK_SIZE/3.

a hackish solution that worked was TASK_UNMAPPED_BASE=TASK_SIZE-0x20000000,
which just assumes that you won't need >512M of mmaped areas.

since the heap grows up and the stack is generally small and limited,
it would be nice to arrange for mmaped areas to grow down.
as far as I can tell, we could just sort vmlist in descending order.  

would there be some problem with doing this?

regards, mark hahn.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Jakub Jelinek]

On Tue, Feb 08, 2000 at 03:08:49PM +0100, Rik van Riel wrote:
> On Tue, 8 Feb 2000, Lee Chin wrote:
> 
> > Sorry if this is the wrong list, but what is the maximum virtual
> > memory an application can malloc in the latest kernel?
> > 
> > Just doing a (for example) "malloc(1024)" in a loop will max out
> > close to 1GB even though I have 4 GB ram on my system.
> 
> The kernel supports up to 3GB of address space per process.
> The first 900MB can be allocated by brk() and the rest can
> be allocated by mmap().
> 
> Problem is that libc malloc() appears to use brk() only, so
> it is limited to 900MB. You can fix that by doing the brk()
> and malloc() yourself, but I think that in the long run the
> glibc people may want to change their malloc implementation
> so that it automatically supports the full 3GB...

glibc malloc is able to use mmap, plus have a lot of tunable things.
See mallopt(3), particularly M_MMAP_THRESHOLD and M_MMAP_MAX parameters.
The default mmap threshold (above which malloc uses mmap) is I think 128K,
but you can decrease it. But it does not make much sense to decrease this
below PAGE_SIZE, because you then waste a lot of memory.

Cheers,
    Jakub
___________________________________________________________________
Jakub Jelinek | jakub@redhat.com | http://sunsite.mff.cuni.cz/~jj
Linux version 2.3.42 on a sparc64 machine (1343.49 BogoMips)
___________________________________________________________________
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Rogier Wolff]

Rik van Riel wrote:
> On Tue, 8 Feb 2000, Lee Chin wrote:
> 
> > Sorry if this is the wrong list, but what is the maximum virtual
> > memory an application can malloc in the latest kernel?
> > 
> > Just doing a (for example) "malloc(1024)" in a loop will max out
> > close to 1GB even though I have 4 GB ram on my system.
> 
> The kernel supports up to 3GB of address space per process.
> The first 900MB can be allocated by brk() and the rest can
> be allocated by mmap().

Another thing to keep in mind is that to allow efficient
allocation/deallocation, there may be some rounding going on. The 2.0
kmalloc would round 1024 to 2048 and therefore waste almost half the
RAM.

> Problem is that libc malloc() appears to use brk() only, so

glibc will use mmap to implement "malloc". libc5 probably uses brk.

> it is limited to 900MB. You can fix that by doing the brk()
> and malloc() yourself, but I think that in the long run the
> glibc people may want to change their malloc implementation
> so that it automatically supports the full 3GB...

			Roger.
-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2137555 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
 "I didn't say it was your fault. I said I was going to blame it on you."
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Eric W. Biederman]

Rik van Riel <riel@nl.linux.org> writes:

> On Tue, 8 Feb 2000, Lee Chin wrote:
> 
> > Sorry if this is the wrong list, but what is the maximum virtual
> > memory an application can malloc in the latest kernel?
> > 
> > Just doing a (for example) "malloc(1024)" in a loop will max out
> > close to 1GB even though I have 4 GB ram on my system.
> 
> The kernel supports up to 3GB of address space per process.
> The first 900MB can be allocated by brk() and the rest can
> be allocated by mmap().
> 
> Problem is that libc malloc() appears to use brk() only, so
> it is limited to 900MB. You can fix that by doing the brk()
> and malloc() yourself, but I think that in the long run the
> glibc people may want to change their malloc implementation
> so that it automatically supports the full 3GB...

Clarification: The problem is the brk interface, which ignores
fragmentation.  The brk interface assumes all memory is
continuous. When brk runs into any mapping it fails. And since ld.so
is mapped at 1GB the brk cannot allocate any more memory.  This
is agravated by the fact that ELF programs appear to be intially
mapped at 128M+288K. 0x08048000. (Someone allocated 900MB??? wow!)

It would certainly be a good option if libc could allocate
new chunks of memory with mmap, or a combination of mmap and mremap.
mremap is functionally a good as brk but will let you work with
arbitrary areas of memory. 

A good option is to compile programs that need huge amounts
of memory through brk statically.  If they do not use mmap, or shmat
they should be fine until they hit the stack, which is growing
in the other direction from 3GB.  Because the program is static it's
code size is reduced the linker will only pull in needed objects,
and performance is also enhanced as you don't need to deal with PIC,
and register starvation.  So it looks good for compute intensive code.

Eric
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Raymond Nijssen]

ebiederm+eric@ccr.net (Eric W. Biederman) wrote:

> Rik van Riel <riel@nl.linux.org> writes:
> 
> > On Tue, 8 Feb 2000, Lee Chin wrote:
> > 
> > > Sorry if this is the wrong list, but what is the maximum virtual
> > > memory an application can malloc in the latest kernel?
> > > 
> > > Just doing a (for example) "malloc(1024)" in a loop will max out
> > > close to 1GB even though I have 4 GB ram on my system.
> > 
> > The kernel supports up to 3GB of address space per process.
> > The first 900MB can be allocated by brk() and the rest can
> > be allocated by mmap().
> > 
> > Problem is that libc malloc() appears to use brk() only, so
> > it is limited to 900MB. You can fix that by doing the brk()
> > and malloc() yourself, but I think that in the long run the
> > glibc people may want to change their malloc implementation
> > so that it automatically supports the full 3GB...
> Clarification: The problem is the brk interface, which ignores
> fragmentation.  The brk interface assumes all memory is
> continuous. When brk runs into any mapping it fails. And since ld.so
> is mapped at 1GB the brk cannot allocate any more memory.  This
> is agravated by the fact that ELF programs appear to be intially
> mapped at 128M+288K. 0x08048000.

I imagine we agree that the problem is not due to brk().
It's due to the fragmented memory map in Linux.


> (Someone allocated 900MB??? wow!)

We use linux systems in compute farms running programs that often need well in
excess of 2GB of memory.  Partly allocated in few large blocks, partly as many
small blocks.

(So the fact that on Linux the kernel wastes the upper 1GB of the address
space does make Solaris/x86 look very attractive, also because it can address
all physical 4GB; but that's a different issue)


> It would certainly be a good option if libc could allocate
> new chunks of memory with mmap, or a combination of mmap and mremap.
> mremap is functionally a good as brk but will let you work with
> arbitrary areas of memory. 

The current implementation of malloc already does that to a limited extent.
The reason why it doesn't go any further than that is because overreliance on
mmap() puts all the burden on the kernel, and you're likely to severely
fragment your memory map.  Remember, mmap has a pagesize resolution.

I have written an extended version that alleviates that problem by jumping
across the shared libs when brk runs into them, but there are some fundamental
problems with this approach that make it work only in well controlled
environments.


So how about getting rid of this memory map dichotomy?

The shared libs could be mapped from 3GB-max_stacksize downwards (rather than
from 1GB upwards).

Is there any reason why this cannot be done?

This would allow brk to grow beyond the 1GB stonewall.


> A good option is to compile programs that need huge amounts
> of memory through brk statically.  If they do not use mmap, or shmat
> they should be fine until they hit the stack, which is growing
> in the other direction from 3GB.  Because the program is static it's
> code size is reduced the linker will only pull in needed objects,
> and performance is also enhanced as you don't need to deal with PIC,
> and register starvation.  So it looks good for compute intensive code.

True.  Unfortunately some libs only come as shared libs.  If your program
needs them, no sigar.


-- 
Raymond Nijssen
raymond@zeropage.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Stephen C. Tweedie]

Hi,

On Sat, Jul 01, 2000 at 10:35:51PM -0700, Raymond Nijssen wrote:

> > It would certainly be a good option if libc could allocate
> > new chunks of memory with mmap, or a combination of mmap and mremap.
> > mremap is functionally a good as brk but will let you work with
> > arbitrary areas of memory. 
> 
> The current implementation of malloc already does that to a limited extent.
> The reason why it doesn't go any further than that is because overreliance on
> mmap() puts all the burden on the kernel, and you're likely to severely
> fragment your memory map.  Remember, mmap has a pagesize resolution.

libc can easily mmap /dev/zero in chunks of multiple MB at a time if
it wants to, and can then dole out that memory as if it was a huge
piece of the heap without kernel involvement.

> So how about getting rid of this memory map dichotomy?
> 
> The shared libs could be mapped from 3GB-max_stacksize downwards (rather than
> from 1GB upwards).
> 
> Is there any reason why this cannot be done?

You then break all programs which allocate large arrays on the stack.

You cannot have an arbitrarily growable heap, AND an arbitrarily
growable stack, AND have the kernel correctly guess where to place
mmaps.

One answer may be to start mmaps down near the heap boundary, and to
teach glibc to be more willing to use mmap() for even small mallocs.
That may break custom malloc libraries but should give the best
results for code which uses the standard glibc malloc: it doesn't
artificially restrain the stack or the mmap area.  Anything
relying directly on [s]brk will be affected, of course.

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Jamie Lokier]

Stephen C. Tweedie wrote:
> libc can easily mmap /dev/zero in chunks of multiple MB at a time if
> it wants to, and can then dole out that memory as if it was a huge
> piece of the heap without kernel involvement.

Several allocators already do that (including GCC's), but Glibc doesn't
AFAIK.

> You cannot have an arbitrarily growable heap, AND an arbitrarily
> growable stack, AND have the kernel correctly guess where to place
> mmaps.
> 
> One answer may be to start mmaps down near the heap boundary, and to
> teach glibc to be more willing to use mmap() for even small mallocs.
> That may break custom malloc libraries but should give the best
> results for code which uses the standard glibc malloc: it doesn't
> artificially restrain the stack or the mmap area.  Anything
> relying directly on [s]brk will be affected, of course.

There are lots of custom malloc libraries.  If you're going to teach
Glibc something anyway, why not add a new mmap flag?

One flag I think would be quite useful is MAP_NOCLOBBER|MAP_FIXED: there
are times when I'd like to be able to _try_ mapping something at a fixed
address, but fail if there is something mapped there already.  Currently
it's necessary to parse /proc/self/maps, and that's far from thread
safe.

The obvious use is for pre-relocated shared libraries, which can run at
any address but will load faster and share more pages if loaded at a
specific address.  (Especially if they're non-PIC).

As a natural extension, a map flag which says "try to map at the
supplied address, but if there is an object there search for a big
enough hole above the supplied address" would simultaneously be useful
for malloc optimisations like you're suggesting, and pre-relocated
shared libraries.  That would be MAP_NOCLOBBER without MAP_FIXED.

-- Jamie
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Stephen C. Tweedie]

Hi,

On Mon, Jul 03, 2000 at 03:32:13PM +0200, Jamie Lokier wrote:

> There are lots of custom malloc libraries.  If you're going to teach
> Glibc something anyway, why not add a new mmap flag?

Because by the time glibc has been loaded, it's too late!  We need the
crt0.o stub to load libdl.so lower in memory or we have already
clobbered the address space irretrievably.  That's why Ben's proposal
for a per-process prctl() for the mmap base would seem to work well
--- you can just use floating mmaps and the kernel will automatically
place them somewhere that won't interfere with really large mallocs.

> One flag I think would be quite useful is MAP_NOCLOBBER|MAP_FIXED: there
> are times when I'd like to be able to _try_ mapping something at a fixed
> address, but fail if there is something mapped there already.

Good idea, but it doesn't help this problem.  

> As a natural extension, a map flag which says "try to map at the
> supplied address, but if there is an object there search for a big
> enough hole above the supplied address" would simultaneously be useful
> for malloc optimisations like you're suggesting, and pre-relocated
> shared libraries.  That would be MAP_NOCLOBBER without MAP_FIXED.

That's what non-MAP_FIXED mmaps already do.  From mmap.c:

	/* Obtain the address to map to. we verify (or select) it and
ensure
	 * that it represents a valid section of the address space.
	 */
	if (flags & MAP_FIXED) {
		if (addr & ~PAGE_MASK)
			return -EINVAL;
	} else {
		addr = get_unmapped_area(addr, len);
		if (!addr)
			return -ENOMEM;
	}

get_unmapped_area looks for the next unmapped area above addr,
with the start address defaulting to TASK_UNMAPPED_BASE only if
addr==0 going in.

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Jamie Lokier]

Stephen C. Tweedie wrote:
> > There are lots of custom malloc libraries.  If you're going to teach
> > Glibc something anyway, why not add a new mmap flag?
> 
> Because by the time glibc has been loaded, it's too late!  We need the
> crt0.o stub to load libdl.so lower in memory or we have already
> clobbered the address space irretrievably.

??  /lib/ld-linux.so.2 is loaded by the kernel so you can easily changed
the way it is mapped.  Everything from there on is part of Glibc.

The only exception is statically linked binaries, and they're not going
to switch to a new malloc anyway.

-- Jamie
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: maximum memory limit

[Raymond Nijssen]

::: "SCT" == Stephen C Tweedie <sct@redhat.com> writes:

 > Hi,
 > On Sat, Jul 01, 2000 at 10:35:51PM -0700, Raymond Nijssen wrote:

 >> > It would certainly be a good option if libc could allocate
 >> > new chunks of memory with mmap, or a combination of mmap and mremap.
 >> > mremap is functionally a good as brk but will let you work with
 >> > arbitrary areas of memory. 
 >> 
 >> The current implementation of malloc already does that to a limited extent.
 >> The reason why it doesn't go any further than that is because overreliance on
 >> mmap() puts all the burden on the kernel, and you're likely to severely
 >> fragment your memory map.  Remember, mmap has a pagesize resolution.

 > libc can easily mmap /dev/zero in chunks of multiple MB at a time if
 > it wants to, and can then dole out that memory as if it was a huge
 > piece of the heap without kernel involvement.

Sure enough, mmapping anonymous chunks is easy.
It is not easy to efficiently manage those chunks.

This fragmentation (these chunks are oftentimes not abutting -- you have
limited control over that) is hard to manage.  Make them big and you'll waste
tons of resources.  Make them small and you'll be unable to consolidate free
ranges to satisfy large requests.

Writing an efficient allocator is actually a pretty tough problem.  It has to
be fast, low-fragmenting, low administration overhead, low waste, etc.
Add to that the ability to co-exist with other sbrk()ing allocators, mmap()ing
routines and other complicating factors.
An erratic context makes those objectives so much harder to achieve.

Yes some workarounds can may be made to work.  They make nices excercises, but
it's still putting the horse behind the carriage.
There's no substitute for a large unadorned memory space.


 >> So how about getting rid of this memory map dichotomy?
 >> 
 >> The shared libs could be mapped from 3GB-max_stacksize downwards (rather than
 >> from 1GB upwards).
 >> 
 >> Is there any reason why this cannot be done?

 > You then break all programs which allocate large arrays on the stack.

Most programmers are wise enough not to assume a large stack.
For one because the stack has a fixed size on all major unices, typically 8MB
(Solaris) or 80MB (HP/UX). 

Does anybody know about programs requiring more than that?


 > You cannot have an arbitrarily growable heap, AND an arbitrarily
 > growable stack, AND have the kernel correctly guess where to place
 > mmaps.

Absolutely.   It's a matter of choosing which one to confine.

Practically it is better to confine the stack.

The rare exception for which the default max stacksize is not enough can be
resolved easily by a wrapper that sets a different resource limit.


 > One answer may be to start mmaps down near the heap boundary, and to
 > teach glibc to be more willing to use mmap() for even small mallocs.
 > That may break custom malloc libraries but should give the best
 > results for code which uses the standard glibc malloc: it doesn't
 > artificially restrain the stack or the mmap area.  Anything
 > relying directly on [s]brk will be affected, of course.

I don't understand this fear for limiting the stack:  in practice that just
isn't a problem.  If anything one should fear fragmenting the heap:  I have
seen many cases in which that is a severe problem.

YMMV of course, and if it does, please share your experiences.

-Raymond
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/