Re: Running out of memory in 1 easy step

Re: Running out of memory in 1 easy step

[Mark_H_Johnson]

I understand about the granularity issue - I was concerned about "extra
pages" above the granularity of allocation.
Thanks also for the sample program - we'll try it with our sample sizes to
make sure we really understand what's being used.
--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of memory in 1 easy step

[Mark Hahn]

> I hate to ask, but where is this behavior described?  I can't find any hint

I was wrong, probably remembering something from old HPUX/etc.
maybe even thinking of libefence.

> of this behavior in the man page. I'm concerned because we would not have
> taken the "guard pages" into account when sizing our application's memory
> usage. We have a real time application where we will also lock the pages

if Linux did guard pages, they would be irrelevant unless you were 
astonishingly naive about using mmap.  glibc, for instance, will mmap
large chunks, and for them, one virtual page is irrelevant.  you'd have
to be making individual O(pagesize) mmaps for this to matter...

> into memory - running out of physical memory in this case is "very bad".

in any case, such (hypothetical on Linux) guard pages would be virtual,
not backed by physical memory.

#include <stdio.h>
#include <sys/mman.h>

char *
mmalloc(unsigned size) {
    char *p = (char*) mmap(0, 
			   size, 
			   PROT_READ|PROT_WRITE, 
			   MAP_PRIVATE|MAP_ANONYMOUS, 
			   0, 
			   0);
    if (p == MAP_FAILED)
	return 0;
    return p;
}
int
main() {
    const unsigned size = 4096;
    char *p = mmalloc(size);
    for (unsigned i=0; i<20; i++) {
	char *n = mmalloc(size);
	printf("next at %p (del %d)\n",n,n-p);
	p = n;
    }
    return 0;
}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of memory in 1 easy step

[Mark_H_Johnson]

I hate to ask, but where is this behavior described?  I can't find any hint
of this behavior in the man page. I'm concerned because we would not have
taken the "guard pages" into account when sizing our application's memory
usage. We have a real time application where we will also lock the pages
into memory - running out of physical memory in this case is "very bad".
Thanks.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>


                                                                                                                                 
                    Mark Hahn                                                                                                    
                    <hahn@coffee.psychology.mc        To:     Wichert Akkerman <wichert@cistron.nl>                              
                    master.ca>                        cc:     linux-mm@kvack.org, (bcc: Mark H Johnson/RTS/Raytheon/US)          
                                                      Subject:     Re: Running out of memory in 1 easy step                      
                    09/14/00 06:03 PM                                                                                            
                                                                                                                                 
                                                                                                                                 



> Not likely, there were still a couple hundreds of megabytes free and
> the process had allocated about 1.5Gb of data.

mmaping 1 to 4096 bytes consumes 8K from your address space:
one for the mmaped page, and one (virtual) guard page (unless
you use MAP_FIXED, of course.)  4G / 8K is approximately the 458878
you reported.  actually, since maps begin at 1G, I would have
expected you to run out sooner...

regards, mark hahn.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of memory in 1 easy step

[Mark Hahn]

> Not likely, there were still a couple hundreds of megabytes free and
> the process had allocated about 1.5Gb of data.

mmaping 1 to 4096 bytes consumes 8K from your address space:
one for the mmaped page, and one (virtual) guard page (unless 
you use MAP_FIXED, of course.)  4G / 8K is approximately the 458878
you reported.  actually, since maps begin at 1G, I would have
expected you to run out sooner...

regards, mark hahn.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of memory in 1 easy step

[Wichert Akkerman]

Previously Andi Kleen wrote:
> The limit is actually 65536 I misremembered it. 
> The main purpose is probably to avoid the counter wrapping. 
> When get_unmapped_area failed you likely just ran out of virtual address space.

Not likely, there were still a couple hundreds of megabytes free and
the process had allocated about 1.5Gb of data.

Wichert.

-- 
  _________________________________________________________________
 /       Nothing is fool-proof to a sufficiently talented fool     \
| wichert@wiggy.net                   http://www.liacs.nl/~wichert/ |
| 1024D/2FA3BC2D 576E 100B 518D 2F16 36B0  2805 3CB8 9250 2FA3 BC2D |
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of memory in 1 easy step

[Andi Kleen]

On Thu, Sep 14, 2000 at 06:08:28PM +0200, Wichert Akkerman wrote:
> Previously Andi Kleen wrote:
> > There is a hardwired limit of 1024 vmas/process. This is to avoid denial
> > of service attacks with attackers using up all memory with vmas.
> 
> That's trivial to circumvent using multiple processes or even threads which
> makes it a useless and possibly damaging protection imho..

The limit is actually 65536 I misremembered it. 
The main purpose is probably to avoid the counter wrapping. 
When get_unmapped_area failed you likely just ran out of virtual address space.


-Andi

-- 
This is like TV. I don't like TV.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of memory in 1 easy step

[Wichert Akkerman]

[-- Attachment #1: Type: text/plain, Size: 615 bytes --]

Previously Andi Kleen wrote:
> There is a hardwired limit of 1024 vmas/process. This is to avoid denial
> of service attacks with attackers using up all memory with vmas.

That's trivial to circumvent using multiple processes or even threads which
makes it a useless and possibly damaging protection imho..

Wichert.

-- 
  _________________________________________________________________
 / Generally uninteresting signature - ignore at your convenience  \
| wichert@liacs.nl                    http://www.liacs.nl/~wichert/ |
| 1024D/2FA3BC2D 576E 100B 518D 2F16 36B0  2805 3CB8 9250 2FA3 BC2D |


[-- Attachment #2: Type: application/pgp-signature, Size: 245 bytes --]

Re: Running out of memory in 1 easy step

[Andi Kleen]

On Thu, Sep 14, 2000 at 03:00:20PM +0200, Wichert Akkerman wrote:
> 
> I have a small test program that consistently can't allocate more
> memory using mmap after 458878 allocations, no matter how much memory
> I allocate per call (tried with 8, 80, 800 and 4000 bytes per call):
> mmap returns ENOMEM. The machine has plenty memory available (2Gb
> and no other processes are running except standard daemons) so there
> should be enough memory.

There is a hardwired limit of 1024 vmas/process. This is to avoid denial
of service attacks with attackers using up all memory with vmas.

-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Running out of memory in 1 easy step

[Wichert Akkerman]

[-- Attachment #1: Type: text/plain, Size: 1402 bytes --]


I have a small test program that consistently can't allocate more
memory using mmap after 458878 allocations, no matter how much memory
I allocate per call (tried with 8, 80, 800 and 4000 bytes per call):
mmap returns ENOMEM. The machine has plenty memory available (2Gb
and no other processes are running except standard daemons) so there
should be enough memory.

Some added printk statements in mm/mmap.c do_mmap_pgoff() revealed
this happens at the following bit of code:

        /* Obtain the address to map to. we verify (or select) it and ensure
         * that it represents a valid section of the address space.
         */
        if (flags & MAP_FIXED) {
                if (addr & ~PAGE_MASK)
                        return -EINVAL;
        } else {
                addr = get_unmapped_area(addr, len);
                if (!addr) {
                        printk("do_mmap_pgoff: cannot allocate unmapped memory, returning ENOMEM\n");
                        return -ENOMEM;
                }
        }

Silly test program is attached (I didn't write it, so don't punish me for
the ugly code :).

Wichert.

-- 
  _________________________________________________________________
 / Generally uninteresting signature - ignore at your convenience  \
| wichert@liacs.nl                    http://www.liacs.nl/~wichert/ |
| 1024D/2FA3BC2D 576E 100B 518D 2F16 36B0  2805 3CB8 9250 2FA3 BC2D |


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mem3.c --]
[-- Type: text/x-csrc, Size: 877 bytes --]

#include <unistd.h>
#include <sys/mman.h>
  
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
int main(){
  int k;
   unsigned int m,p;
   const int mem=800000000;
   int len;
   double **dptr;

   m=sizeof(double)+sizeof(double*);
   len=mem/m;
printf("Will allocate %d bytes in %d blocks\n", mem, len);
   dptr = (double **) malloc(sizeof(double*)*len) ;
   for(k=0; k<len; k++) {
     dptr[k] = (double *) mmap(NULL, 500*sizeof(double), PROT_READ|PROT_WRITE, (MAP_PRIVATE|MAP_ANONYMOUS), -1, 0);
     if(dptr[k]==MAP_FAILED) {
	printf("Allocation error: %s\n", strerror(errno));
       printf("k: %d\n",k);
       printf("Alloc Null\n");
getchar();
       exit(-1);
     }
     *(dptr[k]) = rand();
   }
   for(k=len-1; k>=0; k--) {
     printf("value: %d",*(dptr[k]));
   }
   fprintf(stderr,"k=%d mem=%d MB\n",k,mem/1000000);
   exit (0);
 }