overcommit stuff

overcommit stuff

[Andrew Morton]

Alan,

running 10,000 tiobench threads I'm showing 23 gigs of
`Commited_AS'.  Is this right?  Those pages are shared,
and if they're not PROT_WRITEable then there's no way in
which they can become unshared?   Seems to be excessively
pessimistic?

Or is 2.5 not up to date?

Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[William Lee Irwin III]

On Sat, Sep 21, 2002 at 04:27:02PM -0700, Andrew Morton wrote:
> Alan,
> running 10,000 tiobench threads I'm showing 23 gigs of
> `Commited_AS'.  Is this right?  Those pages are shared,
> and if they're not PROT_WRITEable then there's no way in
> which they can become unshared?   Seems to be excessively
> pessimistic?
> Or is 2.5 not up to date?
> Thanks.

Hmm, that sounds different from what I see, I usually see between
75GB and 250GB of Committed_AS. OTOH that does seem "over the top",
esp. since the ZONE_NORMAL OOM's usually hit with between 18GB and
30GB of ZONE_HIGHMEM totally untouched.


Cheers,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Martin J. Bligh]

> running 10,000 tiobench threads I'm showing 23 gigs of
> `Commited_AS'.  Is this right?  Those pages are shared,
> and if they're not PROT_WRITEable then there's no way in
> which they can become unshared?   Seems to be excessively
> pessimistic?
> 
> Or is 2.5 not up to date?

It's also a global atomic counter that burns up a fair amount
of CPU time bouncing cachelines on the NUMA boxes ... even when 
overcommit is set to 1, and it's not used for anything other 
than meminfo ... any chance of this either becoming a per-cpu 
thing, or dying, or not being used when overcommit is 1?

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Andrew Morton]

"Martin J. Bligh" wrote:
> 
> > running 10,000 tiobench threads I'm showing 23 gigs of
> > `Commited_AS'.  Is this right?  Those pages are shared,
> > and if they're not PROT_WRITEable then there's no way in
> > which they can become unshared?   Seems to be excessively
> > pessimistic?
> >
> > Or is 2.5 not up to date?
> 
> It's also a global atomic counter that burns up a fair amount
> of CPU time bouncing cachelines on the NUMA boxes ... even when
> overcommit is set to 1, and it's not used for anything other
> than meminfo ... any chance of this either becoming a per-cpu
> thing, or dying, or not being used when overcommit is 1?

"It" being vm_committed_space.

The problem is that it's read from frequently, as well as
updated frequently.  So we would still have problems when
we have to reach across and fish the cpu-local counters
out of remote corners of the machine all the time.

The usual tricks for amortising this counter's cost have (serious)
accuracy implications.

I am planning on sitting down and working out exactly what we're
trying to account here - presumably there's another way.  Just
havent got onto it yet.

Worst come to worst, we can hide it inside CONFIG_NOT_WHACKOMATIC
I guess.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Martin J. Bligh]

> "It" being vm_committed_space.
> 
> The problem is that it's read from frequently, as well as
> updated frequently.  So we would still have problems when
> we have to reach across and fish the cpu-local counters
> out of remote corners of the machine all the time.

Not if you set overcommit = 1, as far as I can see.

> The usual tricks for amortising this counter's cost have (serious)
> accuracy implications.

Well, seems it's a rough guess anyway ... at least it's vastly
inaccurate in one direction (pessimistic).
 
> I am planning on sitting down and working out exactly what we're
> trying to account here - presumably there's another way.  Just
> havent got onto it yet.
> 
> Worst come to worst, we can hide it inside CONFIG_NOT_WHACKOMATIC
> I guess.

I was thinking of moving the update in vm_enough_memory under
the switch for what type of overcommit you had, and doing something
similar for the other places it's updated. I suppose that would do
unfortunate things if you turned overcommit from 1 to something
else whilst the system was running though ... not convinced that's
a good idea anyway OTOH.

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Hugh Dickins]

On Sat, 21 Sep 2002, Martin J. Bligh wrote:
> 
> > The usual tricks for amortising this counter's cost have (serious)
> > accuracy implications.
> 
> Well, seems it's a rough guess anyway ... at least it's vastly
> inaccurate in one direction (pessimistic).

Yesss.  I don't think it matters much if it's somewhat inaccurate
(the half-of-memory thing is just pulled out of a hat anyway, isn't
it? and there's no accounting for taste^Hthe kernel's memory usage,
just a hope that it won't go over half).

But it would be very wrong to introduce any indeterminacy in the
calculations, such that the numbers might progressively drift
further and further away from what's right.  That's one of the
reasons it ends up so pessimistic, because it would be impossible
(or too costly) to do the accounting otherwise.

> I was thinking of moving the update in vm_enough_memory under
> the switch for what type of overcommit you had, and doing something
> similar for the other places it's updated. I suppose that would do
> unfortunate things if you turned overcommit from 1 to something
> else whilst the system was running though ... not convinced that's
> a good idea anyway OTOH.

It is intended that you should be able to switch commit modes while
running.  There is one hole there that we've not got around to
plugging yet, the handling of MAP_NORESERVE, but otherwise I believe
it makes sense: please don't take that away.

I like to see those Committed_AS numbers (though I don't care for
the "_AS" prefix), even though I run loose.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Martin J. Bligh]

> It is intended that you should be able to switch commit modes while
> running.  There is one hole there that we've not got around to
> plugging yet, the handling of MAP_NORESERVE, but otherwise I believe
> it makes sense: please don't take that away.

Not quite sure why you'd want to do that, but if you really do, 
I guess making a config option to disable this stuff is a possibility.

> I like to see those Committed_AS numbers (though I don't care for
> the "_AS" prefix), even though I run loose.

Well, there are cheaper ways of keeping it if it's just a stat for
meminfo ;-)

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Hugh Dickins]

On Sat, 21 Sep 2002, Andrew Morton wrote:
> Alan,
> 
> running 10,000 tiobench threads I'm showing 23 gigs of
> `Commited_AS'.  Is this right?  Those pages are shared,
> and if they're not PROT_WRITEable then there's no way in
> which they can become unshared?   Seems to be excessively
> pessimistic?
> 
> Or is 2.5 not up to date?

I don't think Alan can be held responsible for errors in the
overcommit stuff rml ported to 2.5 and I then added fixes to.

I believe it is up to date in 2.5.

Committed_AS certainly errs on the pessimistic side, that's
what it's about.  How much swap do you have i.e. is 23GB
committed impossible, or just surprising to you?  Does the
number go back to what it started off from when you kill
off the tests?  How are "those pages" allocated e.g. what
mmap args?

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Andrew Morton]

Hugh Dickins wrote:
> 
> On Sat, 21 Sep 2002, Andrew Morton wrote:
> > Alan,
> >
> > running 10,000 tiobench threads I'm showing 23 gigs of
> > `Commited_AS'.  Is this right?  Those pages are shared,
> > and if they're not PROT_WRITEable then there's no way in
> > which they can become unshared?   Seems to be excessively
> > pessimistic?
> >
> > Or is 2.5 not up to date?
> 
> I don't think Alan can be held responsible for errors in the
> overcommit stuff rml ported to 2.5 and I then added fixes to.

Well I'm not saying it's an error.  It may be by design.

> I believe it is up to date in 2.5.

OK.
 
> Committed_AS certainly errs on the pessimistic side, that's
> what it's about.  How much swap do you have i.e. is 23GB
> committed impossible, or just surprising to you?  Does the
> number go back to what it started off from when you kill
> off the tests?  How are "those pages" allocated e.g. what
> mmap args?

I have 7G physical, 4G swap.

"those pages" were just used by some scruffy perl script 
running `./tiotest &' ten thousand times.  I assume it's
shared executable text.

It seems very unlikely (impossible?) that those pages will
ever become unshared.

Are they returned when the threads are killed?  Dunno - the
machine got a vists from the NMI watchdog in the scheduler
somewhere before I could tell.  Retesting...

Here's what I had when it died:

MemTotal:      7249608 kB
MemFree:          7180 kB
MemShared:           0 kB
Buffers:         29040 kB
Cached:        6879180 kB
SwapCached:      51216 kB
Active:          22672 kB
Inactive:      6950548 kB
HighTotal:     6422528 kB
HighFree:         2980 kB
LowTotal:       827080 kB
LowFree:          4200 kB
SwapTotal:     3951844 kB
SwapFree:      3829764 kB
Dirty:          658468 kB
Writeback:       10640 kB
Mapped:          57228 kB
Slab:            83188 kB
Committed_AS: 28417140 kB
PageTables:      58152 kB
ReverseMaps:     28455


nr_dirty 165433
nr_writeback 2663 
nr_pagecache 1739859
nr_page_table_pages 14538
nr_reverse_maps 28455
nr_mapped 14307
nr_slab 20802
pswpin 45
pswpout 30532
pgalloc 6671454
pgfree 6673245
pgactivate 74265
pgdeactivate 68457
pgfault 1261681
pgmajfault 714
pgscan 4640872
pgrefill 100136
pgsteal 4329474
kswapd_steal 1413013
pageoutrun 90269
allocstall 90269



       buffer_head:    25669KB    30953KB   82.92
       task_struct:    14648KB    15218KB   96.25
   radix_tree_node:    12337KB    12354KB   99.85
  ext2_inode_cache:     4058KB     4058KB  100.0 
    vm_area_struct:     2463KB     2475KB   99.52
          size-512:     1428KB     1428KB  100.0 
              filp:     1301KB     1301KB  100.0 
      dentry_cache:     1290KB     1290KB  100.0 
biovec-BIO_MAX_PAGES:      780KB      780KB  100.0 
       names_cache:      744KB      748KB   99.46
         biovec-64:      677KB      723KB   93.57
   blkdev_requests:      625KB      633KB   98.69
         size-4096:      556KB      556KB  100.0 
         pte_chain:      158KB      489KB   32.39
sgpool-MAX_PHYS_SEGMENTS:      420KB      480KB   87.50
        biovec-128:      390KB      390KB  100.0 
         size-2048:      352KB      352KB  100.0 
         size-1024:      348KB      348KB  100.0 
           size-32:      317KB      324KB   97.57
  ext3_inode_cache:      171KB      240KB   71.22
         sgpool-64:      213KB      232KB   91.93
        signal_act:      212KB      212KB  100.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Hugh Dickins]

On Sat, 21 Sep 2002, Andrew Morton wrote:
> Hugh Dickins wrote:
> > On Sat, 21 Sep 2002, Andrew Morton wrote:
> > >
> > > running 10,000 tiobench threads I'm showing 23 gigs of
> > > `Commited_AS'.  Is this right?  Those pages are shared,
> > > and if they're not PROT_WRITEable then there's no way in
> > > which they can become unshared?   Seems to be excessively
> > > pessimistic?
>  
> > Committed_AS certainly errs on the pessimistic side, that's
> > what it's about.  How much swap do you have i.e. is 23GB
> > committed impossible, or just surprising to you?  Does the
> > number go back to what it started off from when you kill
> > off the tests?  How are "those pages" allocated e.g. what
> > mmap args?
> 
> I have 7G physical, 4G swap.

When I wondered if impossible, of course I was overlooking
that you wouldn't be running with strict commit limitation,
so "impossible" is quite difficult to reach.

> "those pages" were just used by some scruffy perl script 
> running `./tiotest &' ten thousand times.  I assume it's
> shared executable text.

When I run tiotest here, /proc/<pid>/maps shows a little over
2MB of rwxp or rw-p areas, all to be counted in Committed_AS.
So 23GB for 10,000 of them sounds reasonable.  You think you
have less PROT_WRITE or less MAP_PRIVATE than I'm seeing?

> It seems very unlikely (impossible?) that those pages will
> ever become unshared.

I expect it's very unlikely (short of application bugs) that
those pages would become unshared; but they have been mapped
in such a way that the process is entitled to unshare them,
therefore they have been counted.  A good example of why
Linux does not impose strict commit accounting, and why
you may choose not to use Alan's strict accounting policy.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Andrew Morton]

Hugh Dickins wrote:
> 
> ...
> > It seems very unlikely (impossible?) that those pages will
> > ever become unshared.
> 
> I expect it's very unlikely (short of application bugs) that
> those pages would become unshared; but they have been mapped
> in such a way that the process is entitled to unshare them,
> therefore they have been counted.  A good example of why
> Linux does not impose strict commit accounting, and why
> you may choose not to use Alan's strict accounting policy.
> 

OK, thanks.   Just checking.

Is glibc mapping executables with PROT_WRITE?  If so,
doesn't that rather devalue the whole overcommit thing?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Hugh Dickins]

On Sat, 21 Sep 2002, Andrew Morton wrote:
> Hugh Dickins wrote:
> > ...
> > > It seems very unlikely (impossible?) that those pages will
> > > ever become unshared.
> > 
> > I expect it's very unlikely (short of application bugs) that
> > those pages would become unshared; but they have been mapped
> > in such a way that the process is entitled to unshare them,
> > therefore they have been counted.  A good example of why
> > Linux does not impose strict commit accounting, and why
> > you may choose not to use Alan's strict accounting policy.
> 
> OK, thanks.   Just checking.
> 
> Is glibc mapping executables with PROT_WRITE?  If so,
> doesn't that rather devalue the whole overcommit thing?

No, it looks like glibc is doing the right thing (mapping the code
readonly and the data+bss readwrite).  And I was wrong to say it's
unlikely those pages would ever become unshared: the four 0.5MB
areas look like typical readwrite private anon allocations.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Andrew Morton]

Hugh Dickins wrote:
> 
> On Sat, 21 Sep 2002, Andrew Morton wrote:
> > Hugh Dickins wrote:
> > > ...
> > > > It seems very unlikely (impossible?) that those pages will
> > > > ever become unshared.
> > >
> > > I expect it's very unlikely (short of application bugs) that
> > > those pages would become unshared; but they have been mapped
> > > in such a way that the process is entitled to unshare them,
> > > therefore they have been counted.  A good example of why
> > > Linux does not impose strict commit accounting, and why
> > > you may choose not to use Alan's strict accounting policy.
> >
> > OK, thanks.   Just checking.
> >
> > Is glibc mapping executables with PROT_WRITE?  If so,
> > doesn't that rather devalue the whole overcommit thing?
> 
> No, it looks like glibc is doing the right thing (mapping the code
> readonly and the data+bss readwrite).  And I was wrong to say it's
> unlikely those pages would ever become unshared: the four 0.5MB
> areas look like typical readwrite private anon allocations.
> 

hm.  That would be two megs of real memory per task?  So maybe
I wasn't running 10000 tasks.  It's hard to say - running ps
with that many processes in the machine appears to take longer
than I have left on this earth.

Maybe an `ls /proc | wc' would tell me.  Dunno; I've moved onto
other bugs for today.  Bill seems to be into this stuff.  Hopefully
he'll retest on the next -mm, which should be a bit nicer to
those-who-run-too-many-tiobenches.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[William Lee Irwin III]

On Sun, Sep 22, 2002 at 12:46:59AM +0100, Hugh Dickins wrote:
> I don't think Alan can be held responsible for errors in the
> overcommit stuff rml ported to 2.5 and I then added fixes to.
> I believe it is up to date in 2.5.
> Committed_AS certainly errs on the pessimistic side, that's
> what it's about.  How much swap do you have i.e. is 23GB
> committed impossible, or just surprising to you?  Does the
> number go back to what it started off from when you kill
> off the tests?  How are "those pages" allocated e.g. what
> mmap args?
> Hugh

In my case it's not really possible to rerun a test in the same
boot. It's not really survived very often, and when it has, it
generally fails to start a second time. Various other things feel the
OOM sting then, e.g. kernel compiles, small task count dbench, etc.

Some of this might be slab, but I think there might be a leak.
The best answers I've come up with thus far are "Hrm, the OOM killer
gets set off at the wrong times, and maybe delalloc would kill bh's?"

Cheers,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: overcommit stuff

[Andrew Morton]

William Lee Irwin III wrote:
> 
> On Sun, Sep 22, 2002 at 12:46:59AM +0100, Hugh Dickins wrote:
> > I don't think Alan can be held responsible for errors in the
> > overcommit stuff rml ported to 2.5 and I then added fixes to.
> > I believe it is up to date in 2.5.
> > Committed_AS certainly errs on the pessimistic side, that's
> > what it's about.  How much swap do you have i.e. is 23GB
> > committed impossible, or just surprising to you?  Does the
> > number go back to what it started off from when you kill
> > off the tests?  How are "those pages" allocated e.g. what
> > mmap args?
> > Hugh
> 
> In my case it's not really possible to rerun a test in the same
> boot. It's not really survived very often, and when it has, it
> generally fails to start a second time. Various other things feel the
> OOM sting then, e.g. kernel compiles, small task count dbench, etc.
> 
> Some of this might be slab, but I think there might be a leak.
> The best answers I've come up with thus far are "Hrm, the OOM killer
> gets set off at the wrong times, and maybe delalloc would kill bh's?"
> 

I see no leak.  After a 30-40 minute run I managed to wrestle all
the threads to the ground, unmounted the target fs and ended
up with


             total       used       free     shared    buffers     cached
Mem:       7249612      62796    7186816          0      11016      10804
-/+ buffers/cache:      40976    7208636
Swap:      3951844         80    3951764
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/