More vm benchmarking

More vm benchmarking

[Nick Piggin]

Well you can imagine my surprise to see your numbers so I've started
redoing some benchmarks to see what is going wrong.

This first set are 2.6.3, 2.6.3-mm2, 2.6.3-mm3. All SMP kernels
compiled with the same compiler and using the same .config (where
possible). Booting with maxcpus=1 and mem=64M. Test is gcc 3.3.3
compiling 2.4.21. I can provide any other information you're
interested in.

While previously I have been doing a single run of a range of
different parallelisation factors, here I've done two runs each over a
smaller range so you can see I am getting fairly consistient results.

kernel | run | -j5 | -j10 | -j15 |
2.6.3    1     136   886    2511
2.6.3    2     150   838    2465

-mm2     1     136   646    1484
-mm2     2     142   676    1265

-mm3     1     135   881    1828
-mm3     2     146   790    1844

This quite clearly shows your patches hurting as I told you. Why did
it get slower? I assume it is because the batching patch places uneven
pressure on normal and DMA zones. This leads to suboptimal eviction
choice - anything else would be a sign of fundamental problems.

Regarding Nikita and my patches, they all showed improvements on this
machine for this type of test *except* the throttling patch which
didn't cause any change. I just thought it was courteous to try not to
stall a possibly unlucky run.

I will now try a set of SMP tests and possibly ones with different
available memory. I would be disappointed but not very surprised if
SMP is causing lots of problems.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Nick Piggin]

Nick Piggin wrote:

> Well you can imagine my surprise to see your numbers so I've started
> redoing some benchmarks to see what is going wrong.
>
> This first set are 2.6.3, 2.6.3-mm2, 2.6.3-mm3. All SMP kernels
> compiled with the same compiler and using the same .config (where


Sorry, the 2.6.3, 2.6.3-mm2 and 2.6.3-mm3 I tested were all UP compiled
kernels.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Andrew Morton]

Nick Piggin <piggin@cyberone.com.au> wrote:
>
> kernel | run | -j5 | -j10 | -j15 |
>  2.6.3    1     136   886    2511
>  2.6.3    2     150   838    2465
> 
>  -mm2     1     136   646    1484
>  -mm2     2     142   676    1265
> 
>  -mm3     1     135   881    1828
>  -mm3     2     146   790    1844
> 
>  This quite clearly shows your patches hurting as I told you.

Probably.  But these differences are small, relative to some differences
wrt 2.4.x

> Why did it get slower?

Dunno.  Maybe the workload prefers imbalanced zone scanning.

> I assume it is because the batching patch places uneven
>  pressure on normal and DMA zones.

The patch improves highmem-vs-lowmem balancing from 10:1 to 1:1.  What
makes you think that it worsens ZONE_NORMAL-vs-ZONE_DMA balancing?

It's easy enough to instrument - just split pgsteal_lo into pgsteal_normal
and pgsteal_dma.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Nick Piggin]

Andrew Morton wrote:

>Nick Piggin <piggin@cyberone.com.au> wrote:
>
>>kernel | run | -j5 | -j10 | -j15 |
>> 2.6.3    1     136   886    2511
>> 2.6.3    2     150   838    2465
>>
>> -mm2     1     136   646    1484
>> -mm2     2     142   676    1265
>>
>> -mm3     1     135   881    1828
>> -mm3     2     146   790    1844
>>
>> This quite clearly shows your patches hurting as I told you.
>>
>
>Probably.  But these differences are small, relative to some differences
>wrt 2.4.x
>
>

2.4 should be pretty close to -mm2 for -j10 and hopefully a
bit worse at -j15. That is what previous benchmarks have been
showing anyway. I better get some 2.4 numbers.

Either way, they're not that small.

>>Why did it get slower?
>>
>
>Dunno.  Maybe the workload prefers imbalanced zone scanning.
>
>

Seriously? I find that a bit hard to swallow. Especially
considering I wouldn't have anything that uses ZONE_DMA
on this system.

>>I assume it is because the batching patch places uneven
>> pressure on normal and DMA zones.
>>
>
>The patch improves highmem-vs-lowmem balancing from 10:1 to 1:1.  What
>makes you think that it worsens ZONE_NORMAL-vs-ZONE_DMA balancing?
>
>It's easy enough to instrument - just split pgsteal_lo into pgsteal_normal
>and pgsteal_dma.
>
>

Sure that can tell you if something is really wrong, but it
is pretty hard to read much from that.

Anyway I don't have code or numbers right now so that means
I have to shut up ;)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Andrew Morton]

Nick Piggin <piggin@cyberone.com.au> wrote:
>
> >Dunno.  Maybe the workload prefers imbalanced zone scanning.
>  >
>  >
> 
>  Seriously? I find that a bit hard to swallow. Especially
>  considering I wouldn't have anything that uses ZONE_DMA
>  on this system.

Could be.  Suppose we have a bunch of really-hard-to-reclaim pages which
only get reclaimed when we're under extreme pressure.  The pages which get
placed there live longer than they should, thus avoiding later periods of
extreme pressure.   Sounds like crap, but it might be true ;)

I'll do the pgsteal_lo splitup.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Andrew Morton]

Andrew Morton <akpm@osdl.org> wrote:
>
> I'll do the pgsteal_lo splitup.

OK, did that.   Running `make -j4 vmlinux' on the 2-way with mem=64m:

Count how many pages were reclaimed from the various zones:

					DMA	NORMAL	HIGH
up to shrink_slab-for-all-zones:	3749	192580	0	(1:51)
up to zone-balancing-fix:		5816	144545		(1:24)
up to zone-balancing-batching:		21446	85209		(1:4)

It should be 1:3, but it's tons better than it used to be.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Nick Piggin]

Andrew Morton wrote:

>Andrew Morton <akpm@osdl.org> wrote:
>
>>I'll do the pgsteal_lo splitup.
>>
>
>OK, did that.   Running `make -j4 vmlinux' on the 2-way with mem=64m:
>
>Count how many pages were reclaimed from the various zones:
>
>					DMA	NORMAL	HIGH
>up to shrink_slab-for-all-zones:	3749	192580	0	(1:51)
>up to zone-balancing-fix:		5816	144545		(1:24)
>up to zone-balancing-batching:		21446	85209		(1:4)
>
>It should be 1:3, but it's tons better than it used to be.
>
>

Yeah I'm not sure if that is entirely true though. You would
expect ZONE_NORMAL to have more pages reclaimed from it
because there should be more pressure on it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Andrew Morton]

Nick Piggin <piggin@cyberone.com.au> wrote:
>
> You would
> expect ZONE_NORMAL to have more pages reclaimed from it
> because there should be more pressure on it.

Why?

The only things which should be special about ZONE_NORMAL which I can think
of are:

a) All the early-allocated pinned memory is sitting there and

b) If you start an app which uses a lot of memory, its text pages will
   probabyl be in ZONE_NORMAL while ZONE_DMA will contain just bss and
   pagecache.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Nick Piggin]

Andrew Morton wrote:

>Nick Piggin <piggin@cyberone.com.au> wrote:
>
>>You would
>>expect ZONE_NORMAL to have more pages reclaimed from it
>>because there should be more pressure on it.
>>
>
>Why?
>
>The only things which should be special about ZONE_NORMAL which I can think
>of are:
>
>a) All the early-allocated pinned memory is sitting there and
>
>b) If you start an app which uses a lot of memory, its text pages will
>   probabyl be in ZONE_NORMAL while ZONE_DMA will contain just bss and
>   pagecache.
>

Maybe. If you're doing a heavy swapping kbuild, stuff will
be pretty randomly placed. And ZONE_NORMAL will just get
more pressure due to trying to allocate from there first.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Andrew Morton]

Nick Piggin <piggin@cyberone.com.au> wrote:
>
> And ZONE_NORMAL will just get
> more pressure due to trying to allocate from there first.

No, that shouldn't be the case.  Once ZONE_NORMAL hits pages_high it just
sits there doing nothing until ZONE_DMA hits pages_high too.  That's the
point at we run (now proportional) page reclaim against both zones.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: More vm benchmarking

[Nick Piggin]

Andrew Morton wrote:

>Nick Piggin <piggin@cyberone.com.au> wrote:
>
>>And ZONE_NORMAL will just get
>>more pressure due to trying to allocate from there first.
>>
>
>No, that shouldn't be the case.  Once ZONE_NORMAL hits pages_high it just
>sits there doing nothing until ZONE_DMA hits pages_high too.  That's the
>point at we run (now proportional) page reclaim against both zones.
>

After it gets past that stage though, and goes into direct
reclaim... Anyway I'll look into it more.

BTW. The SMP kbuild benchmarks are looking much the same
as the UP ones just with a little bit more variance. Our
tests must be running in parallel universes or something :(

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>