Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Andrew Morton]

bugme-daemon@osdl.org wrote:
>
> http://bugme.osdl.org/show_bug.cgi?id=3268
> 
>            Summary: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb
>     Kernel Version:  2.6.8.1-mm4
>             Status: NEW
>           Severity: high
>              Owner: akpm@digeo.com
>          Submitter: kmannth@us.ibm.com
>                 CC: mbligh@aracnet.com
> 
> 
> Distribution:  SuSE SLES9 base
> Hardware Environment:  IBM x445 8-way  32gb and 16 gb 
> Software Environment:   2.6.8.1-mm4
> Problem Description:  I run out of lowmemory very easily using /dev/shm/
> I have 64g and Numa/Discontig enabled in my kernel.  
> 
> Steps to reproduce:  Fill up 1/2 or more of /dev/shm (on my system it is about
> 1/3-1/2 of my total system memory) with lots of kernel builds.  Observe system
> breakdown.  (If you want the script I will email it to you).  I have seen this
> with both 32 gigs and 16 gigs...
> 
> For example if I boot with 16gb and I start 45 kernel builds in /dev/shm they
> system take about 30-60 seconds to run out of lowmemory.  
> The oom killer comes in and starts shutting down processes.  The system is unsuable.
>   There is tons of highmem left but no lowmem.  Is there something about
> /dev/shm that might cause this? 
> 
> I turned some various vm debug optoins that printed out some info that I will
> attach.  I will attach the config file and as much of the kernel messages as I
> can.

I assume this is because we're using up all of lowmem with filesystem metadata.

Hugh?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[keith]

On Tue, 2004-08-24 at 14:43, Andrew Morton wrote:
> bugme-daemon@osdl.org wrote:
> >
> > http://bugme.osdl.org/show_bug.cgi?id=3268
> > 
> >            Summary: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb
> >     Kernel Version:  2.6.8.1-mm4
> >             Status: NEW
> >           Severity: high
> >              Owner: akpm@digeo.com
> >          Submitter: kmannth@us.ibm.com
> >                 CC: mbligh@aracnet.com
> > 
> > 
> > Distribution:  SuSE SLES9 base
> > Hardware Environment:  IBM x445 8-way  32gb and 16 gb 
> > Software Environment:   2.6.8.1-mm4
> > Problem Description:  I run out of lowmemory very easily using /dev/shm/
> > I have 64g and Numa/Discontig enabled in my kernel.  
> > 
> > Steps to reproduce:  Fill up 1/2 or more of /dev/shm (on my system it is about
> > 1/3-1/2 of my total system memory) with lots of kernel builds.  Observe system
> > breakdown.  (If you want the script I will email it to you).  I have seen this
> > with both 32 gigs and 16 gigs...
> > 
> > For example if I boot with 16gb and I start 45 kernel builds in /dev/shm they
> > system take about 30-60 seconds to run out of lowmemory.  
> > The oom killer comes in and starts shutting down processes.  The system is unsuable.
> >   There is tons of highmem left but no lowmem.  Is there something about
> > /dev/shm that might cause this? 
> > 
> > I turned some various vm debug optoins that printed out some info that I will
> > attach.  I will attach the config file and as much of the kernel messages as I
> > can.
> 
> I assume this is because we're using up all of lowmem with filesystem metadata.
> 
> Hugh?
> 

As a note. I am able to run this test with Reiser just fine.  It looks
like the shmfs code always calls kmem_cache_alloc when it needs to
allocate a new inode.  
  If the kmem_cache_alloc calls are filling up lowmem shouldn't the
alloc calls fail eventually instead of sending out the oom killer to all
the processes in the system?  Is it possible to fail an kmem_cache_alloc
call?
  
Keith Mannthey

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Hugh Dickins]

On Tue, 24 Aug 2004, Andrew Morton wrote:
> bugme-daemon@osdl.org wrote:
> >
> > http://bugme.osdl.org/show_bug.cgi?id=3268
> > 
> >            Summary: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb
> > Problem Description:  I run out of lowmemory very easily using /dev/shm/
> > I have 64g and Numa/Discontig enabled in my kernel.  
> > 
> > Steps to reproduce:  Fill up 1/2 or more of /dev/shm (on my system it is about
> > 1/3-1/2 of my total system memory) with lots of kernel builds.  Observe system
> > breakdown.  (If you want the script I will email it to you).  I have seen this
> > with both 32 gigs and 16 gigs...
> 
> I assume this is because we're using up all of lowmem with filesystem metadata.
> 
> Hugh?

Probably, though it's not something anyone reported before.

Filesystem metadata being, not tmpfs's indirect blocks (which use highmem),
but the plenitude of inodes and dentries: which would get pruned if it were
a disk-based filesystem, but cannot because this one is all in memory.

I've not done the arithmetic... tomorrow.  I'll try to reproduce something
similar (I don't have 16GB and I don't have NUMA, though latter probably
not relevant) tomorrow, and fix (decide default nr_inodes by lowmem).

Keith, please do mail me your script (in case there's something special
in there e.g. will make a difference if your sources are linked or not),
and also your /proc/slabinfo near OOMing, if that's convenient.

Thanks,
Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Andrew Morton]

keith <kmannth@us.ibm.com> wrote:
>
> Is it possible to fail an kmem_cache_alloc
> call?

Normally not, in the current VM setup (Andrea has differences
of opinion on this, so the code is set up to switch policies).

The kmem_cache_alloc() caller can change that policy by adding
in __GFP_NORETRY.

We'll know more when you've added meminfo/slabinfo/buddyinfo.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Hugh Dickins]

On Wed, 25 Aug 2004, keith wrote:
> With the patch I only get 82294 inodes.  Hmmmm.... I don't have any
> lowmem issues but I can't create too many files.  

I get 107701 for my LowTotal 861608 kB, but your LowTotal is 658808 kB.

I'm not saying that default (in your case 82294) is the highest allowable,
just a safe default, consistent with what was offered before when no
highmem.  If the default is to allow half of RAM to be occupied by tmpfs
(highmem and swappable) data blocks, it seems appropriate to default to
a much smaller fraction of lowmem for the unswappable metadata; but I
may be overdoing that, I'm not sure.

I expect in your case (not wanting the lowmem for anything else much)
you'll survive with nr_inodes=400000: you can try that, and push it up
to breaking point if you like (one reason I don't want to enforce an
upper limit in the kernel, just a default).  But I don't expect you to
survive with nr_inodes=700000.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Martin J. Bligh]

>> Well, by default it does give you a maximum number of inodes for tmpfs:
>> which at 16GB of RAM (in 4KB pages: odd that that factors in, but I've
>> no great urge to depart from existing defaults except where it's buggy)
>> would give you 2M inodes.  But since each might roughly be expected to
>> consume 1KB (a shmem_inode, a dentry, a longer name, a radix node) of
>> low memory, we'd want 2GB of lowmem to support all those: too much.
> 
> I think there is a need for a hard upper limit to the number of inodes
> tmpfs can use.  With this system I can have 32gb of memory and I also
> have a 64gb(i386) system. tempfs sizes above 15=g are definitely
> possible.  What about a 40gig tmpfs.... 

The problem is ... how much mem are you willing to lose? Anything that's
unshrinkable is very, very dangerous ... whatever the limit you put on it.
But I agree, it'd be better than the current.

>> So, how well does the patch below work for you, if you leave nr_inodes
>> to its default?  Carry on setting size=15G, that should be okay; but
>> I don't think the kernel should stop you if you insist on setting
>> nr_inodes to something unfortunate like 10M.  "df -i" will show
>> how many inodes it's actually giving you by default.
> 
> I was wondering how to do that thanks. 
> With the patch I only get 82294 inodes.  Hmmmm.... I don't have any
> lowmem issues but I can't create too many files.  

Well, you have to pick one or the other ;-)

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[keith]

On Wed, 2004-08-25 at 13:37, Hugh Dickins wrote:
> On Wed, 25 Aug 2004, keith wrote:
> > 
> > I turned CONFIG_DEBUG_PAGEALLOC off.
> 
> Good, thanks.
> 
> > I ran into problems when I tried
> > building 60 kernel trees or so.  It is using about 10gb of tmpfs space. 
> > See http://bugme.osdl.org/attachment.cgi?id=3566&action=view for more
> > info.
> 
> Okay, that's more to the point.
> 
> > It looks like there should be a maximum number of inodes for tmpfs.
> > Because I didn't know how many inodes it is using but I gave it a large
> > playground to use (mount -t tmpfs -o size=15G,nr_inodes=10000k,mode=0700
> > tmpfs /mytmpfs)
> 
> Well, by default it does give you a maximum number of inodes for tmpfs:
> which at 16GB of RAM (in 4KB pages: odd that that factors in, but I've
> no great urge to depart from existing defaults except where it's buggy)
> would give you 2M inodes.  But since each might roughly be expected to
> consume 1KB (a shmem_inode, a dentry, a longer name, a radix node) of
> low memory, we'd want 2GB of lowmem to support all those: too much.

I think there is a need for a hard upper limit to the number of inodes
tmpfs can use.  With this system I can have 32gb of memory and I also
have a 64gb(i386) system. tempfs sizes above 15=g are definitely
possible.  What about a 40gig tmpfs.... 
> 
> So, how well does the patch below work for you, if you leave nr_inodes
> to its default?  Carry on setting size=15G, that should be okay; but
> I don't think the kernel should stop you if you insist on setting
> nr_inodes to something unfortunate like 10M.  "df -i" will show
> how many inodes it's actually giving you by default.

I was wondering how to do that thanks. 
With the patch I only get 82294 inodes.  Hmmmm.... I don't have any
lowmem issues but I can't create too many files.  

Thanks,
  Keith Mannthey 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Andrew Morton]

"Martin J. Bligh" <mbligh@aracnet.com> wrote:
>
> > Hugh Dickins <hugh@veritas.com> wrote:
> >> 
> >> (hmm, does lowmem shortage exert
> >>  any pressure on highmem cache these days, I wonder?);
> > 
> > It does, indirectly - when we reclaim an unused inode we also shoot down
> > all that inode's pagecache.
> 
> ISTR that causes some fairly major problems under mem pressure - when we
> go to shrink inode cache, it used to sit there for *ages* trying to free
> pagecache, particularly if there were a lot of large dirty files. That
> was a few months back ...

I never saw a report.

> but would anything have fixed that case since then?
> 

No.  Probably testing I_DIRTY earlier in prune_icache() would fix it up.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Martin J. Bligh]

> Hugh Dickins <hugh@veritas.com> wrote:
>> 
>> (hmm, does lowmem shortage exert
>>  any pressure on highmem cache these days, I wonder?);
> 
> It does, indirectly - when we reclaim an unused inode we also shoot down
> all that inode's pagecache.

ISTR that causes some fairly major problems under mem pressure - when we
go to shrink inode cache, it used to sit there for *ages* trying to free
pagecache, particularly if there were a lot of large dirty files. That
was a few months back ... but would anything have fixed that case since then?

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Hugh Dickins]

On Wed, 25 Aug 2004, Andrew Morton wrote:
> Hugh Dickins <hugh@veritas.com> wrote:
> >
> > (hmm, does lowmem shortage exert
> >  any pressure on highmem cache these days, I wonder?);
> 
> It does, indirectly - when we reclaim an unused inode we also shoot down
> all that inode's pagecache.

Not much help in the tmpfs case, where the inode cannot get to be unused
without being deleted.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Andrew Morton]

Hugh Dickins <hugh@veritas.com> wrote:
>
> (hmm, does lowmem shortage exert
>  any pressure on highmem cache these days, I wonder?);

It does, indirectly - when we reclaim an unused inode we also shoot down
all that inode's pagecache.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Hugh Dickins]

On Wed, 25 Aug 2004, keith wrote:
> 
> I turned CONFIG_DEBUG_PAGEALLOC off.

Good, thanks.

> I ran into problems when I tried
> building 60 kernel trees or so.  It is using about 10gb of tmpfs space. 
> See http://bugme.osdl.org/attachment.cgi?id=3566&action=view for more
> info.

Okay, that's more to the point.

> It looks like there should be a maximum number of inodes for tmpfs.
> Because I didn't know how many inodes it is using but I gave it a large
> playground to use (mount -t tmpfs -o size=15G,nr_inodes=10000k,mode=0700
> tmpfs /mytmpfs)

Well, by default it does give you a maximum number of inodes for tmpfs:
which at 16GB of RAM (in 4KB pages: odd that that factors in, but I've
no great urge to depart from existing defaults except where it's buggy)
would give you 2M inodes.  But since each might roughly be expected to
consume 1KB (a shmem_inode, a dentry, a longer name, a radix node) of
low memory, we'd want 2GB of lowmem to support all those: too much.

So, how well does the patch below work for you, if you leave nr_inodes
to its default?  Carry on setting size=15G, that should be okay; but
I don't think the kernel should stop you if you insist on setting
nr_inodes to something unfortunate like 10M.  "df -i" will show
how many inodes it's actually giving you by default.

Andrew, this paragraph is more for you...
If you examine my 1KB calculation above, you may conclude that this
won't be quite the final patch: so long as there's ample swap, the
(more than 1) radix nodes shouldn't be a problem since they would
melt away under lowmem pressure (hmm, does lowmem shortage exert
any pressure on highmem cache these days, I wonder?); but doesn't
link(1) imply that dentries can take up almost unlimited space?
Looks as if tmpfs ought to be limiting links as it limits inodes.

Hugh

--- 2.6.8.1-mm4/mm/shmem.c	2004-08-23 12:20:31.000000000 +0100
+++ linux/mm/shmem.c	2004-08-25 20:47:27.072015312 +0100
@@ -1818,9 +1818,10 @@
 
 	/*
 	 * Per default we only allow half of the physical ram per
-	 * tmpfs instance
+	 * tmpfs instance; limiting inodes to 1 per 2 pages of lowmem.
 	 */
 	blocks = inodes = totalram_pages / 2;
+	inodes -= totalhigh_pages / 2;
 
 #ifdef CONFIG_TMPFS
 	if (shmem_parse_options(data, &mode, &uid, &gid, &blocks, &inodes)) {

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[keith]

On Wed, 2004-08-25 at 07:05, Hugh Dickins wrote:
> On Tue, 24 Aug 2004, keith wrote:
> > 
> > Ok I created an attachment in the bug for the slab/buddy/mem info.  
> > You can watch zone normal get exhausted :)
> > http://bugme.osdl.org/show_bug.cgi?id=3268
> 
> Thanks.  Yes, your lowmem is full of Slab, and that's entirely
> unsurprising since you have CONFIG_DEBUG_PAGEALLOC on: so every
> slab object needs a full 4096-byte page to itself (well, there
> are some exceptions, but that doesn't change the picture).
> 
> That's a _very_ distorting config option, and I think this means that
> your report is of no interest in itself - sorry.  But it does raise a
> valid question whether it can happen in real, non-debug life - thanks.

I turned CONFIG_DEBUG_PAGEALLOC off.  I ran into problems when I tried
building 60 kernel trees or so.  It is using about 10gb of tmpfs space. 
See http://bugme.osdl.org/attachment.cgi?id=3566&action=view for more
info.

The bug also contains the config file used.  
> 
> I'll do the arithmetic on that when I've more leisure: I expect the
> answer to be that it can happen, and I ought to adjust defaulting of
> maximum tmpfs inodes.

It looks like there should be a maximum number of inodes for tmpfs.
Because I didn't know how many inodes it is using but I gave it a large
playground to use (mount -t tmpfs -o size=15G,nr_inodes=10000k,mode=0700
tmpfs /mytmpfs)

Thanks,
  Keith Mannthey 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Bug 3268] New: Lowmemory exhaustion problem with v2.6.8.1-mm4 16gb

[Hugh Dickins]

On Tue, 24 Aug 2004, keith wrote:
> 
> Ok I created an attachment in the bug for the slab/buddy/mem info.  
> You can watch zone normal get exhausted :)
> http://bugme.osdl.org/show_bug.cgi?id=3268

Thanks.  Yes, your lowmem is full of Slab, and that's entirely
unsurprising since you have CONFIG_DEBUG_PAGEALLOC on: so every
slab object needs a full 4096-byte page to itself (well, there
are some exceptions, but that doesn't change the picture).

That's a _very_ distorting config option, and I think this means that
your report is of no interest in itself - sorry.  But it does raise a
valid question whether it can happen in real, non-debug life - thanks.

I'll do the arithmetic on that when I've more leisure: I expect the
answer to be that it can happen, and I ought to adjust defaulting of
maximum tmpfs inodes.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>