VM Report was:Re: Break 2.4 VM in five easy steps

VM Report was:Re: Break 2.4 VM in five easy steps

[Shane Nay]

(VM report at Marcelo Tosatti's request.  He has mentioned that rather than 
complaining about the VM that people mention what there experiences were.  I 
have tried to do so in the way that he asked.)

> 1) Describe what you're running. (your workload)

A lot of daemons, all on a private network so there is no throughput load on 
them.  About 13 rxvt's, freeamp actively playing music at all times, xemacs 
with 25 active buffers, a few instances of vi, opera, no "desktop env", just 
windowmaker.  (Though I have a few KDE2 apps open, and one or two GTK based 
apps open, so lots of library code swapping in and out I imagine)  Now what 
I've noticed lately is this, with 2.4.2 my machine would lock quite 
frequently when I was compiling code and had other apps that were allocing 
memory.  With 2.4.5 I haven't had that behaviour, but I've been much lighter 
on my machine.  (I was doing full toolchain builds with 2.4.2 when I had the 
real problems)  But processes were still running when the machine would 
lock..., like the mp3 player was still playing I noticed one time.  With 
2.4.5 (not -ac) I haven't had any deadlocks, but the system seems very 
sluggish at acute moments .  While doing absolutely nothing processor 
intensive (I've been loading up top and ps'ing with regularity when this 
happens, looking for kswapd going crazy), when I switch between workspaces 
the refresh is much more sluggish on occasion, like I can watch windows 
appear.  Almost like a micro freeze really.
(AMD T-Bird 1.333Mhz 256MB-DDR)

> 2) Describe what you're feeling. (eg "interactivity is crap when I run
> this or that thing", etc)

Freeing memory takes *forever*, but I think that's a function of how I'm 
allocing in this polygon rendering routine I'm working on.  Like literally 
sucks up vast numbers of cycles and makes picogui totally unusable.  But I 
think this is unrelated to the kernel..., I think that's just because I 
haven't implemented re-use in memory structures for the polygon routine.  
(It's malloc/freeing massive numbers of small chunks of memory rather than 
doing it's own memory management, probably related to glibc memory 
organization)

Here's a vmstat line after a 8 days of uptime and before contrived mem tests:
   procs                      memory    swap          io     system         
cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  
id
 1  0  0      0   3056   7856 121872   0   0     7     4   37    16   1   0  
40


> If we need more info than that I'll request in private.
>
> Also send this reports to the linux-mm list, so other VM hackers can also
> get those reports and we avoid traffic on lk.

> By performance you mean interactivity or throughput?

Interactivity.  I don't have any throughput needs to speak of.

I just ran a barage of tests on my machine, and the smallest it would ever 
make the cache was 16M, it would prefer to kill processes rather than make 
the cache smaller than that.

Contrived stressor program: (pseudo code)
fork(); fork(); fork(); fork();  //16 total processes
for (i=0;i<n;i++) {
  ptr=malloc(1M)
  while(++m<ptrsize) ptr[m]='b';
  sleep(2);
}
I would change n such that the total amount of memory was less than the 
amount of cache plus free memory.  Running this put the entire system into 
chaos in short order.  After it had killed off only one of the contrived 
memory hungry processes and at least two others (MP3 player and opera), the 
machine was slugish..., very slow to respond to any key input.  It stayed in 
this near freeze state for about 20 seconds, after that it started to speed 
up to user input gradually.  (Probably swapping code from disk into cache or 
something like that)  It took about 5-10 minutes to come back "up to speed".

> Just do what I described above.

Done :).

Thanks,
Shane Nay.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Jonathan Morton]

At 12:29 am +0100 8/6/2001, Shane Nay wrote:
>(VM report at Marcelo Tosatti's request.  He has mentioned that rather than
>complaining about the VM that people mention what there experiences were.  I
>have tried to do so in the way that he asked.)

>> By performance you mean interactivity or throughput?
>
>Interactivity.  I don't have any throughput needs to speak of.
>
>I just ran a barage of tests on my machine, and the smallest it would ever
>make the cache was 16M, it would prefer to kill processes rather than make
>the cache smaller than that.

http://www.chromatix.uklinux.net/linux-patches/vm-update-2.patch

Try this.  I can't guarantee it's SMP-safe yet (I'm leaving the gurus to
that, but they haven't told me about any errors in the past hour so I'm
assuming they aren't going to find anything glaringly wrong...), but you
might like to see if your performance improves with it.  It also fixes the
OOM-killer bug, which you refer to above.

Some measurements, from my own box (1GHz Athlon, 256Mb RAM):

For the following benchmarks, physical memory availability was reduced
according to the parameter in the left column.  The benchmark is the
wall-clock time taken to compile MySQL.

mem=	2.4.5		earlier tweaks	now
48M	8m30s		6m30s		5m58s
32M	unknown		2h15m		12m34s

The following was performed with all 256Mb RAM available.  This is
compilation of MySQL using make -j 15.

kernel:		2.4.5		now
time:		6m30s		6m15s
peak swap:	190M		70M

For the following test, the 256Mb swap partition on my IDE drive was
disabled and replaced with a 1Gb swapfile on my Ultra160 SCSI drive.  This
is compilation of MySQL using make -j 20.

kernel:		2.4.5		now
time:		7m20s		6m30s
peak swap:	370M		254M

Draw your own conclusions.  :)

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)

The key to knowledge is not to rely on people to teach you it.

GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Mike Galbraith]

On Fri, 8 Jun 2001, Jonathan Morton wrote:

> http://www.chromatix.uklinux.net/linux-patches/vm-update-2.patch
>
> Try this.  I can't guarantee it's SMP-safe yet (I'm leaving the gurus to
> that, but they haven't told me about any errors in the past hour so I'm
> assuming they aren't going to find anything glaringly wrong...), but you
> might like to see if your performance improves with it.  It also fixes the
> OOM-killer bug, which you refer to above.
>
> Some measurements, from my own box (1GHz Athlon, 256Mb RAM):
>
> For the following benchmarks, physical memory availability was reduced
> according to the parameter in the left column.  The benchmark is the
> wall-clock time taken to compile MySQL.
>
> mem=	2.4.5		earlier tweaks	now
> 48M	8m30s		6m30s		5m58s
> 32M	unknown		2h15m		12m34s
>
> The following was performed with all 256Mb RAM available.  This is
> compilation of MySQL using make -j 15.
>
> kernel:		2.4.5		now
> time:		6m30s		6m15s
> peak swap:	190M		70M
>
> For the following test, the 256Mb swap partition on my IDE drive was
> disabled and replaced with a 1Gb swapfile on my Ultra160 SCSI drive.  This
> is compilation of MySQL using make -j 20.
>
> kernel:		2.4.5		now
> time:		7m20s		6m30s
> peak swap:	370M		254M
>
> Draw your own conclusions.  :)

(ok;)

Hi,

I gave this a shot at my favorite vm beater test (make -j30 bzImage)
while testing some other stuff today.

seven identical runs, six slightly different kernels plus yours.

real    11m23.522s  2.4.5.vm-update-2
user    7m59.170s
sys     0m37.030s
user  :       0:08:07.06  65.6%  page in :   642402
nice  :       0:00:00.00   0.0%  page out:   676820
system:       0:02:09.44  17.4%  swap in :   105965
idle  :       0:02:05.66  16.9%  swap out:   162603

real    10m9.512s  2.4.5.virgin
user    7m55.520s
sys     0m35.460s
user  :       0:08:02.66  72.2%  page in :   535186
nice  :       0:00:00.00   0.0%  page out:   377992
system:       0:01:37.78  14.6%  swap in :    99445
idle  :       0:01:28.14  13.2%  swap out:    81926

real    10m48.939s 2.4.5.virgin+reclaim.marcelo
user    7m54.960s
sys     0m36.240s
user  :       0:08:02.33  68.0%  page in :   566239
nice  :       0:00:00.00   0.0%  page out:   431874
system:       0:01:56.02  16.4%  swap in :   108633
idle  :       0:01:50.61  15.6%  swap out:    96415

real    9m54.466s 2.4.5.virgin+reclaim.mike (icky 'bleeder valve')
user    7m57.370s
sys     0m35.890s
user  :       0:08:04.74  74.1%  page in :   527678
nice  :       0:00:00.00   0.0%  page out:   405259
system:       0:01:12.01  11.0%  swap in :    98616
idle  :       0:01:37.47  14.9%  swap out:    91492

real    9m12.198s  2.4.5.tweak
user    7m41.290s
sys     0m34.840s
user  :       0:07:47.69  76.8%  page in :   452632
nice  :       0:00:00.00   0.0%  page out:   399847
system:       0:01:17.08  12.7%  swap in :    75338
idle  :       0:01:03.97  10.5%  swap out:    88291

real    9m41.563s  2.4.5.tweak+reclaim.marcelo
user    7m59.880s
sys     0m34.690s
user  :       0:08:07.22  73.4%  page in :   515433
nice  :       0:00:00.00   0.0%  page out:   545762
system:       0:01:35.34  14.4%  swap in :    88425
idle  :       0:01:21.11  12.2%  swap out:   125967

real    9m47.682s  2.4.5.tweak+reclaim.mike
user    8m2.190s
sys     0m34.550s
user  :       0:08:09.57  75.7%  page in :   513166
nice  :       0:00:00.00   0.0%  page out:   473539
system:       0:01:20.27  12.4%  swap in :    83127
idle  :       0:01:16.89  11.9%  swap out:   108886

Conclusion:

Your patch hits the cache too hard and pays through the nose for
doing so.. at least under this hefty weight load it does.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Tobias Ringstrom]

On Fri, 8 Jun 2001, Mike Galbraith wrote:
> I gave this a shot at my favorite vm beater test (make -j30 bzImage)
> while testing some other stuff today.

Could you please explain what is good about this test?  I understand that
it will stress the VM, but will it do so in a realistic and relevant way?

Isn't the interesting case when you have a number of processes using lots
of memory, but only a part of all that memory is beeing actively used, and
that memory fits in RAM.  In that case, the VM should make sure that the
not used memory is swapped out.  In RAM you should have the used memory,
but also disk cache if there is any RAM left.  Does the current VM handle
this case fine yet?  IMHO, this is the case most people care about.  It is
definately the case I care about, at least. :-)

I'm not saying that it's a completely uninteresting case when your active
memory is bigger than you RAM of course, but perhaps there should be other
algorithms handling that case, such as putting some of the swapping
processes to sleep for some time, especially if you have lots of processes
competing for the memory. I may be wrong, but it seems to me that your
testcase falls into this second category (also known as thrashing).

An at last, a humble request:  Every problem I've had with the VM has been
that it either swapped out too many processes and used too much cache, or
the other way around.  I'd really enjoy a way to tune this behaviour, if
possible.

/Tobias

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[John Stoffel]

>>>>> "Tobias" == Tobias Ringstrom <tori@unhappy.mine.nu> writes:

Tobias> On Fri, 8 Jun 2001, Mike Galbraith wrote:

>> I gave this a shot at my favorite vm beater test (make -j30 bzImage)
>> while testing some other stuff today.

Tobias> Could you please explain what is good about this test?  I
Tobias> understand that it will stress the VM, but will it do so in a
Tobias> realistic and relevant way?

I agree, this isn't really a good test case.  I'd rather see what
happens when you fire up a gimp session to edit an image which is
*almost* the size of RAM, or even just 50% the size of ram.  Then how
does that affect your other processes that are running at the same
time?  

This testing could even be automated with the script-foo stuff to get
consistent results across runs, which is the prime requirement of any
sort of testing.  

On another issue, in swap.c we have two defines for buffer_mem and
page_cache, but the first maxes out at 60%, while the cache maxes out
at 75%.  Shouldn't they both be lower numbers?  Or at least equally
sized?

I've set my page_cache maximum to be 60, I'll be trying to test it
over the weekend, but good weather will keep me outside doing other
stuff...

Thanks,
John
   John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
	 stoffel@lucent.com - http://www.lucent.com - 978-952-7548
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Mike Galbraith]

On Fri, 8 Jun 2001, John Stoffel wrote:

> >>>>> "Tobias" == Tobias Ringstrom <tori@unhappy.mine.nu> writes:
>
> Tobias> On Fri, 8 Jun 2001, Mike Galbraith wrote:
>
> >> I gave this a shot at my favorite vm beater test (make -j30 bzImage)
> >> while testing some other stuff today.
>
> Tobias> Could you please explain what is good about this test?  I
> Tobias> understand that it will stress the VM, but will it do so in a
> Tobias> realistic and relevant way?
>
> I agree, this isn't really a good test case.  I'd rather see what
> happens when you fire up a gimp session to edit an image which is
> *almost* the size of RAM, or even just 50% the size of ram.  Then how
> does that affect your other processes that are running at the same
> time?

OK, riddle me this.  If this test is a crummy test, just how is it
that I was able to warn Rik in advance that when 2.4.5 was released,
he should expect complaints?  How did I _know_ that?  The answer is
that I fiddle with Rik's code a lot, and I test with this test because
it tells me a lot.  It may not tell you anything, but it does me.

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[John Stoffel]

Mike> OK, riddle me this.  If this test is a crummy test, just how is
Mike> it that I was able to warn Rik in advance that when 2.4.5 was
Mike> released, he should expect complaints?  How did I _know_ that?
Mike> The answer is that I fiddle with Rik's code a lot, and I test
Mike> with this test because it tells me a lot.  It may not tell you
Mike> anything, but it does me.

I never said it was a crummy test, please do not read more into my
words than was written.  What I was trying to get across is that just
one test (such as a compile of the kernel) isn't perfect at showing
where the problems are with the VM sub-system.

Jonathan Morton has been using another large compile to also test the
sub-system, and it includes a compile which puts a large, single
process pressure on the VM.  I consider this to be a more
representative test of how the VM deals with pressure.  

The kernel compile is an ok test of basic VM handling, but from what
I've been hearing on linux-kernel and linux-mm is that the VM goes to
crap when you have a mix of stuff running, and one (or more) processes
starts up or grows much larger and starts impacting the system
performance.

I'm also not knocking your contributions to this discussion, so stop
being so touchy.  I was trying to contribute and say (albeit poorly)
that a *mix* of tests is needed to test the VM.

More importantly, a *repeatable* set of tests is what is needed to
test the VM and get consistent results from run to run, so you can see
how your changes are impacting performance.  The kernel compile
doesn't really have any one process grow to a large fraction of
memory, so dropping in a compile which *does* is a good thing.

John
   John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
	 stoffel@lucent.com - http://www.lucent.com - 978-952-7548

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Marcelo Tosatti]

On Fri, 8 Jun 2001, John Stoffel wrote:

> 
> Mike> OK, riddle me this.  If this test is a crummy test, just how is
> Mike> it that I was able to warn Rik in advance that when 2.4.5 was
> Mike> released, he should expect complaints?  How did I _know_ that?
> Mike> The answer is that I fiddle with Rik's code a lot, and I test
> Mike> with this test because it tells me a lot.  It may not tell you
> Mike> anything, but it does me.
> 
> I never said it was a crummy test, please do not read more into my
> words than was written.  What I was trying to get across is that just
> one test (such as a compile of the kernel) isn't perfect at showing
> where the problems are with the VM sub-system.
> 
> Jonathan Morton has been using another large compile to also test the
> sub-system, and it includes a compile which puts a large, single
> process pressure on the VM.  I consider this to be a more
> representative test of how the VM deals with pressure.  
> 
> The kernel compile is an ok test of basic VM handling, but from what
> I've been hearing on linux-kernel and linux-mm is that the VM goes to
> crap when you have a mix of stuff running, and one (or more) processes
> starts up or grows much larger and starts impacting the system
> performance.
> 
> I'm also not knocking your contributions to this discussion, so stop
> being so touchy.  I was trying to contribute and say (albeit poorly)
> that a *mix* of tests is needed to test the VM.
> 
> More importantly, a *repeatable* set of tests is what is needed to
> test the VM and get consistent results from run to run, so you can see
> how your changes are impacting performance.  The kernel compile
> doesn't really have any one process grow to a large fraction of
> memory, so dropping in a compile which *does* is a good thing.

I agree with you. 

Mike, I'm sure you have noticed that stock kernel gives much better
results than mine or Jonathan's patch.

Now the stock kernel gives us crappy interactivity compared to my patch.
(Note: my patch still does not gives me the interactivity I want under
high VM loads, but I hope to get there soon).

BTW, we are talking with the OSDL (http://www.osdlab.org) guys about a
possibility to setup a test system which would run a different variety of
benchmarks to give us results of different kinds of workloads. If that
ever happens, we'll probably get rid of most of this testing problems.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[John Stoffel]

Marcelo> Now the stock kernel gives us crappy interactivity compared
Marcelo> to my patch.  (Note: my patch still does not gives me the
Marcelo> interactivity I want under high VM loads, but I hope to get
Marcelo> there soon).

This raises the important question, how can we objectively measure
interactive response in the kernel and relate it to the user's
perceived interactive response?  If we could come up with some sort of
testing system that would show us this, it would help alot, since we
could just have people run tests in a more automatic and repeatable
manner.

And I think it would also help us automatically tune the Kernel, since
it would have a knowledge of it's own performance.  

There is the problem in terms of some people want pure interactive
performance, while others are looking for throughput over all else,
but those are both extremes of the spectrum.  Though I suspect
raw throughput is the less wanted (in terms of numbers of systems)
than keeping interactive response good during VM pressure.

I have zero knowledge of how we could do this, but giving the kernel
some counters, even if only for use during debugging runs, which would
give us some objective feedback on performance would be a big win.

Having people just send in reports of "I ran X,Y,Z and it was slow"
doesn't help us, since it's so hard to re-create their environment so
you can run tests against it.

Anyway, enjoy the weekend all.

John
   John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
	 stoffel@lucent.com - http://www.lucent.com - 978-952-7548
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Marcelo Tosatti]

On Fri, 8 Jun 2001, John Stoffel wrote:

> 
> Marcelo> Now the stock kernel gives us crappy interactivity compared
> Marcelo> to my patch.  (Note: my patch still does not gives me the
> Marcelo> interactivity I want under high VM loads, but I hope to get
> Marcelo> there soon).
> 
> This raises the important question, how can we objectively measure
> interactive response in the kernel and relate it to the user's
> perceived interactive response?  If we could come up with some sort of
> testing system that would show us this, it would help alot, since we
> could just have people run tests in a more automatic and repeatable
> manner.
> 
> And I think it would also help us automatically tune the Kernel, since
> it would have a knowledge of it's own performance.  
> 
> There is the problem in terms of some people want pure interactive
> performance, while others are looking for throughput over all else,
> but those are both extremes of the spectrum.  Though I suspect
> raw throughput is the less wanted (in terms of numbers of systems)
> than keeping interactive response good during VM pressure.

And this raises a very very important point: raw throughtput wins
enterprise-like benchmarks, and the enterprise people are the ones who pay
most of hackers here. (including me and Rik)

We have to be careful about that. 

> I have zero knowledge of how we could do this, but giving the kernel
> some counters, even if only for use during debugging runs, which would
> give us some objective feedback on performance would be a big win.
> 
> Having people just send in reports of "I ran X,Y,Z and it was slow"
> doesn't help us, since it's so hard to re-create their environment so
> you can run tests against it.

Lets wait for some test system to be set up (eg the OSDL thing).

Once thats done, I'm sure we will find out some way of doing it. 

Well, good weekend for you too. 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Jonathan Morton]

[ Re-entering discussion after too long a day and a long sleep... ]

>> There is the problem in terms of some people want pure interactive
>> performance, while others are looking for throughput over all else,
>> but those are both extremes of the spectrum.  Though I suspect
>> raw throughput is the less wanted (in terms of numbers of systems)
>> than keeping interactive response good during VM pressure.
>
>And this raises a very very important point: raw throughtput wins
>enterprise-like benchmarks, and the enterprise people are the ones who pay
>most of hackers here. (including me and Rik)

Very true.  As well as the fact that interactivity is much harder to
measure.  The question is, what is interactivity (from the kernel's
perspective)?  It usually means small(ish) processes with intermittent
working-set and CPU requirements.  These types of process can safely be
swapped out when not immediately in use, but the kernel has to be able to
page them in quite quickly when needed.  Doing that under heavy load is
very non-trivial.

It can also mean multimedia applications with a continuous (maybe small)
working set, a continuous but not 100% CPU usage, and the special property
that the user WILL notice if this process gets swapped out even briefly.
mpg123 and XMMS fall into this category, and I sometimes tried running
these alongside my compilation tests to see how they fared.  I think I had
it going fairly well towards the end, with mpg123 stuttering relatively
rarely and briefly while VM load was high.

On the subject of Mike Galbraith's kernel compilation test, how much
physical RAM does he have for his machine, what type of CPU is it, and what
(approximate) type of device does he use for swap?  I'll see if I can
partially duplicate his results at this end.  So far all my tests have been
done with a fast CPU - perhaps I should try the P166/MMX or even try
loading linux-pmac onto my 8100.

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)

The key to knowledge is not to rely on people to teach you it.

GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Andrew Morton]

Jonathan Morton wrote:
> 
> [ Re-entering discussion after too long a day and a long sleep... ]
> 
> >> There is the problem in terms of some people want pure interactive
> >> performance, while others are looking for throughput over all else,
> >> but those are both extremes of the spectrum.  Though I suspect
> >> raw throughput is the less wanted (in terms of numbers of systems)
> >> than keeping interactive response good during VM pressure.
> >
> >And this raises a very very important point: raw throughtput wins
> >enterprise-like benchmarks, and the enterprise people are the ones who pay
> >most of hackers here. (including me and Rik)
> 
> Very true.  As well as the fact that interactivity is much harder to
> measure.  The question is, what is interactivity (from the kernel's
> perspective)?  It usually means small(ish) processes with intermittent
> working-set and CPU requirements.  These types of process can safely be
> swapped out when not immediately in use, but the kernel has to be able to
> page them in quite quickly when needed.  Doing that under heavy load is
> very non-trivial.

For the low-latency stuff, latency can be defined as
the worst-case time to schedule a userspace process in
response to an interrupt.

That metric is also appropriate in this case, (latency equals
interactivity), although here you don't need to be so fanatical
about the *worst case*.  A few scheduling blips here are less
fatal.

I have tools to measure latency (aka interactivity).  At
http://www.uow.edu.au/~andrewm/linux/schedlat.html#downloads
there is a kernel patch called `rtc-debug' which causes
the PC RTC to generate a stream of interrupts.  A user-space
task called `amlat' responds to those interrupts and
reads the RTC device.  The patched RTC driver can then
measure the elapsed time between the interrupt and the
read from userspace.  Voila: latency.

When you close the RTC device (by killing amlat), the RTC
driver will print out a histogram of the latencies.

`amlat' at present gives itself SCHED_RR policy and
runs under mlockall() - for your testing you'll need
to delete those lines.

So.  Simple apply rtc-debug, run `amlat' and kill it
when you've finished the workload.

The challenge will be to relate the latency histogram
to human-perceived interactivity.   I'm not sure of
the best way of doing that.  Perhaps monitor the 90th
percentile, and aim to keep it below 100 milliseconds.
Also, `amlat' should do a bit of disk I/O as well.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Mike Galbraith]

On Sat, 9 Jun 2001, Jonathan Morton wrote:

> On the subject of Mike Galbraith's kernel compilation test, how much
> physical RAM does he have for his machine, what type of CPU is it, and what
> (approximate) type of device does he use for swap?  I'll see if I can
> partially duplicate his results at this end.  So far all my tests have been
> done with a fast CPU - perhaps I should try the P166/MMX or even try
> loading linux-pmac onto my 8100.

It's a PIII/500 with one ide disk.

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Jonathan Morton]

>> On the subject of Mike Galbraith's kernel compilation test, how much
>> physical RAM does he have for his machine, what type of CPU is it, and what
>> (approximate) type of device does he use for swap?  I'll see if I can
>> partially duplicate his results at this end.  So far all my tests have been
>> done with a fast CPU - perhaps I should try the P166/MMX or even try
>> loading linux-pmac onto my 8100.
>
>It's a PIII/500 with one ide disk.

...with how much RAM?  That's the important bit.

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)

The key to knowledge is not to rely on people to teach you it.

GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Mike Galbraith]

On Sat, 9 Jun 2001, Jonathan Morton wrote:

> >> On the subject of Mike Galbraith's kernel compilation test, how much
> >> physical RAM does he have for his machine, what type of CPU is it, and what
> >> (approximate) type of device does he use for swap?  I'll see if I can
> >> partially duplicate his results at this end.  So far all my tests have been
> >> done with a fast CPU - perhaps I should try the P166/MMX or even try
> >> loading linux-pmac onto my 8100.
> >
> >It's a PIII/500 with one ide disk.
>
> ...with how much RAM?  That's the important bit.

Duh! :) I'm a dipstick.  128mb.

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Mike Galbraith]

On Fri, 8 Jun 2001, Marcelo Tosatti wrote:

> On Fri, 8 Jun 2001, John Stoffel wrote:
>
> > More importantly, a *repeatable* set of tests is what is needed to
> > test the VM and get consistent results from run to run, so you can see
> > how your changes are impacting performance.  The kernel compile
> > doesn't really have any one process grow to a large fraction of
> > memory, so dropping in a compile which *does* is a good thing.
>
> I agree with you.
>
> Mike, I'm sure you have noticed that stock kernel gives much better
> results than mine or Jonathan's patch.

I noticed that Jonathan brought back waiting.. that (among others)
made me veeeeery interested.

> Now the stock kernel gives us crappy interactivity compared to my patch.
> (Note: my patch still does not gives me the interactivity I want under
> high VM loads, but I hope to get there soon).

(And that's why)  Among other things (yes, I do love throughput) I've
poked at the interactivity problem. I can't improve it anymore without
doing some strategic waiting :(  I used to be able to help it a little
by doing a careful roll-up in scrub size as load builds.. trying to
smooth the transition from latency oriented to hammer down throughput.

> BTW, we are talking with the OSDL (http://www.osdlab.org) guys about a
> possibility to setup a test system which would run a different variety of
> benchmarks to give us results of different kinds of workloads. If that
> ever happens, we'll probably get rid of most of this testing problems.

Excellent!

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Mike Galbraith]

On Fri, 8 Jun 2001, John Stoffel wrote:

> Mike> OK, riddle me this.  If this test is a crummy test, just how is
> Mike> it that I was able to warn Rik in advance that when 2.4.5 was
> Mike> released, he should expect complaints?  How did I _know_ that?
> Mike> The answer is that I fiddle with Rik's code a lot, and I test
> Mike> with this test because it tells me a lot.  It may not tell you
> Mike> anything, but it does me.
>
> I never said it was a crummy test, please do not read more into my
> words than was written.  What I was trying to get across is that just
> one test (such as a compile of the kernel) isn't perfect at showing
> where the problems are with the VM sub-system.

Hmm...

Tobias> Could you please explain what is good about this test?  I
Tobias> understand that it will stress the VM, but will it do so in a
Tobias> realistic and relevant way?

I agree, this isn't really a good test case.  I'd rather see what
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
happens when you fire up a gimp session to edit an image which is
*almost* the size of RAM, or even just 50% the size of ram.  Then how
does that affect your other processes that are running at the same
time?

...but anyway, yes it just one test from any number of possibles.

> Jonathan Morton has been using another large compile to also test the
> sub-system, and it includes a compile which puts a large, single
> process pressure on the VM.  I consider this to be a more
> representative test of how the VM deals with pressure.

What does 'more representative' mean given that the VM must react to
every situation it runs into?

> The kernel compile is an ok test of basic VM handling, but from what

Now we're communicating.  I never said it was more than that ;-)

> I've been hearing on linux-kernel and linux-mm is that the VM goes to
> crap when you have a mix of stuff running, and one (or more) processes
> starts up or grows much larger and starts impacting the system
> performance.
>
> I'm also not knocking your contributions to this discussion, so stop
> being so touchy.  I was trying to contribute and say (albeit poorly)
> that a *mix* of tests is needed to test the VM.

Yes, more people need to test. I don't need to do all of those other
tests (no have right toys), more people need to do repeatable tests.

> More importantly, a *repeatable* set of tests is what is needed to
> test the VM and get consistent results from run to run, so you can see
> how your changes are impacting performance.  The kernel compile
> doesn't really have any one process grow to a large fraction of
> memory, so dropping in a compile which *does* is a good thing.

I know I'm only watching basic functionality.  I'm watching basic
functionality with one very consistant test run very consistantly.

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Zlatko Calusic]

Mike Galbraith <mikeg@wen-online.de> writes:

> On Fri, 8 Jun 2001, John Stoffel wrote:
> 
> > Mike> OK, riddle me this.  If this test is a crummy test, just how is
> > Mike> it that I was able to warn Rik in advance that when 2.4.5 was
> > Mike> released, he should expect complaints?  How did I _know_ that?
> > Mike> The answer is that I fiddle with Rik's code a lot, and I test
> > Mike> with this test because it tells me a lot.  It may not tell you
> > Mike> anything, but it does me.
> >
> > I never said it was a crummy test, please do not read more into my
> > words than was written.  What I was trying to get across is that just
> > one test (such as a compile of the kernel) isn't perfect at showing
> > where the problems are with the VM sub-system.
> 
> Hmm...
> 
> Tobias> Could you please explain what is good about this test?  I
> Tobias> understand that it will stress the VM, but will it do so in a
> Tobias> realistic and relevant way?
> 
> I agree, this isn't really a good test case.  I'd rather see what
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> happens when you fire up a gimp session to edit an image which is
> *almost* the size of RAM, or even just 50% the size of ram.  Then how
> does that affect your other processes that are running at the same
> time?
> 
> ...but anyway, yes it just one test from any number of possibles.

One great test that I'm using regularly to see what's goin' on, is at
http://lxr.linux.no/. It is a cool utility to cross reference your
Linux kernel source tree, and in the mean time eat gobs of memory, do
lots of I/O, and burn many CPU cycles (all at the same time). Ideal
test, if you ask me and if anybody has the time, it would be nice to
see different timing numbers when run on different kernels. Just make
sure you run it on the same kernel tree to make reproducable results.
It has three passes, and the third one is the most interesting one
(use vmstat 1 to see why). When run with 64MB RAM configuration, it
would swap heavily, with 128MB somewhat, and at 192MB maybe not
(depending on the other applications running at the same time).

Try it, it is a nice utility, and a great test. :)
-- 
Zlatko
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Rik van Riel]

On Fri, 8 Jun 2001, Mike Galbraith wrote:
> On Fri, 8 Jun 2001, John Stoffel wrote:

> > I agree, this isn't really a good test case.  I'd rather see what
> > happens when you fire up a gimp session to edit an image which is
> > *almost* the size of RAM, or even just 50% the size of ram.
> 
> OK, riddle me this.  If this test is a crummy test, just how is it

Personally, I'd like to see BOTH of these tests, and many many
more.

Preferably, handed to the VM hackers in various colourful
graphs that allow even severely undercaffeinated hackers to
see how things changed for the good or the bad between kernel
revisions.

cheers,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Mike Galbraith]

On Fri, 8 Jun 2001, Tobias Ringstrom wrote:

> On Fri, 8 Jun 2001, Mike Galbraith wrote:
> > I gave this a shot at my favorite vm beater test (make -j30 bzImage)
> > while testing some other stuff today.
>
> Could you please explain what is good about this test?  I understand that
> it will stress the VM, but will it do so in a realistic and relevant way?

Can you explain what is bad about this test? ;)  It spins the same VM wheels
as any other load does.  What's the difference if I have a bunch of httpd
allocating or a bunch of cc1/as/ld?  This load has a modest cachable data
set and is compute bound.. and above all gives very repeatable results.

I use it to watch reaction to surge.  I watch for the vm to build to a
solid maximum throughput without thrashing.  That's the portion of VM
that I'm interested in, so that's what I test.  Besides :) I simply don't
have the hardware to try to simulate hairy chested server loads.  There
are lots of folks with hairy chested boxes.. they should test that stuff.

I've been repeating ~this test since 2.0 times, and have noticed a 1:1
relationship.  When I notice that my box is ~happy doing this load test,
I also notice very few VM gripes hitting the list.

> Isn't the interesting case when you have a number of processes using lots
> of memory, but only a part of all that memory is beeing actively used, and
> that memory fits in RAM.  In that case, the VM should make sure that the
> not used memory is swapped out.  In RAM you should have the used memory,
> but also disk cache if there is any RAM left.  Does the current VM handle
> this case fine yet?  IMHO, this is the case most people care about.  It is
> definately the case I care about, at least. :-)

The interesting case is _every_ case.  Try seeing my particular test as
a simulation of a small classroom box with 30 students compiling their
assignments and it'll suddenly become quite realistic.  You'll notice
by the numbers I post that I was very careful to not overload the box in
a rediculous manner when selecting the total size of the job.. it's just
a heavily loaded box.  This test does not overload my IO resources, so
it tests the VM's ability to choose and move the right stuff at the right
time to get the job done with a minimum of additional overhead.

The current VM handles things generally well imho, but has problems
regulating itself under load.  My test load hits the VM right in it's
weakest point (not _that_ weak, but..) by starting at zero and building
rapidly to max.. and keeping it _right there_.

> I'm not saying that it's a completely uninteresting case when your active
> memory is bigger than you RAM of course, but perhaps there should be other
> algorithms handling that case, such as putting some of the swapping
> processes to sleep for some time, especially if you have lots of processes
> competing for the memory. I may be wrong, but it seems to me that your
> testcase falls into this second category (also known as thrashing).

Thrashing?  Let's look some numbers. (not the ugly ones, the ~ok ones;)

real    9m12.198s  make -j 30 bzImage
user    7m41.290s
sys     0m34.840s
user  :       0:07:47.69  76.8%  page in :   452632
nice  :       0:00:00.00   0.0%  page out:   399847
system:       0:01:17.08  12.7%  swap in :    75338
idle  :       0:01:03.97  10.5%  swap out:    88291

real    8m6.994s  make bzImage
user    7m34.350s
sys     0m26.550s
user  :       0:07:37.52  78.4%  page in :    90546
nice  :       0:00:00.00   0.0%  page out:    18164
system:       0:01:26.13  14.8%  swap in :        1
idle  :       0:00:39.69   6.8%  swap out:        0

...look at cpu utilization.  One minute +tiny change to complete the
large job vs the small (VM footprint) job.

The box is not thrashing, it's working it's little silicon butt off.
What I'm testing is the VM's ability to handle load without thrashing
so badly that it loses throughput bigtime, stalls itself whatever..
it's ability to regulate itself.  I consider a minute and a half to
be ~acceptable, a minute to be good, and 30 seconds to be excellent.
That's just my own little VM performance thermometer.

> An at last, a humble request:  Every problem I've had with the VM has been
> that it either swapped out too many processes and used too much cache, or
> the other way around.  I'd really enjoy a way to tune this behaviour, if
> possible.

Tunables aren't really practical in VM (imho).  If there were a dozen
knobs, you'd have to turn a dozen knobs a dozen times a day.  VM has
to be self regulating.

In case you can't tell (the length of this reply) I like my fovorite
little generic throughput test a LOT :-)

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Tobias Ringstrom]

On Fri, 8 Jun 2001, Mike Galbraith wrote:
> On Fri, 8 Jun 2001, Tobias Ringstrom wrote:
> > On Fri, 8 Jun 2001, Mike Galbraith wrote:
> > > I gave this a shot at my favorite vm beater test (make -j30 bzImage)
> > > while testing some other stuff today.
> >
> > Could you please explain what is good about this test?  I understand that
> > it will stress the VM, but will it do so in a realistic and relevant way?
>
> Can you explain what is bad about this test? ;)  It spins the same VM wheels

I think a load of ~30 is quit uncommon, and therefor it is unclear to me
that it would be a test that would be repesentative of most normal loads.

> as any other load does.  What's the difference if I have a bunch of httpd
> allocating or a bunch of cc1/as/ld?  This load has a modest cachable data
> set and is compute bound.. and above all gives very repeatable results.

Not a big difference.  The difference I was thinking abount is the
difference between spawning lots of processes allocating, using and
freeing lots of memory, compared to a case where you have a few processes
touching a lot of already allocated pages in some pattern.  I was
wondering whether optimizing for your case would be good or bad for the
other case.  I know, I know, I should do more testing myself.  And I
should probably not ask you, since you really really like your test,
and you will probably just say yes... ;-)

At home, I'm running a couple of computers.  One of them is a slow
computer running Linux, serving mail, NFS, SMB, etc.  I'm usually logged
in on a couple of virtual consoles.  On this machine, I do not mind if all
shells, daemons and other idle processes are beeing swapped out in favor
of disk cache for the NFS and SMB serving.  In fact, that is a very good
thing, and I want it that way.

Another maching is my desktop machine.  When using this maching, I really
hate when my emacsen, browsers, xterms, etc are swapped out just to give
me some stupid disk cache for my xmms or compilations.  I do not care if a
kernel compile is a little slower as long as my applications are snappy.

How could Linux predict this?  It is a matter of taste, IMHO.

> I use it to watch reaction to surge.  I watch for the vm to build to a
> solid maximum throughput without thrashing.  That's the portion of VM
> that I'm interested in, so that's what I test.  Besides :) I simply don't
> have the hardware to try to simulate hairy chested server loads.  There
> are lots of folks with hairy chested boxes.. they should test that stuff.

Agreed.  More testing is needed.  Now if we would have those knobs and
wheels to turn, we could perhaps also tune our systems to behave as we
like them, and submit that as well.  Right now you need to be a kernel
hacker, and see through all the magic with shm, mmap, a bunch of caches,
page lists, etc.  I'd give a lot for a nice picture (or state diagram)
showing the lifetime of a page, but I have not found such a picture
anywhere.  Besides, the VM seems to change every new release anyway.

> I've been repeating ~this test since 2.0 times, and have noticed a 1:1
> relationship.  When I notice that my box is ~happy doing this load test,
> I also notice very few VM gripes hitting the list.

Ok, but as you say, we need more tests.

> > Isn't the interesting case when you have a number of processes using lots
> > of memory, but only a part of all that memory is beeing actively used, and
> > that memory fits in RAM.  In that case, the VM should make sure that the
> > not used memory is swapped out.  In RAM you should have the used memory,
> > but also disk cache if there is any RAM left.  Does the current VM handle
> > this case fine yet?  IMHO, this is the case most people care about.  It is
> > definately the case I care about, at least. :-)
>
> The interesting case is _every_ case.  Try seeing my particular test as
> a simulation of a small classroom box with 30 students compiling their
> assignments and it'll suddenly become quite realistic.  You'll notice
> by the numbers I post that I was very careful to not overload the box in
> a rediculous manner when selecting the total size of the job.. it's just
> a heavily loaded box.  This test does not overload my IO resources, so
> it tests the VM's ability to choose and move the right stuff at the right
> time to get the job done with a minimum of additional overhead.

I did not understand those numbers when I saw them the first time.  Now, I
must say that your test does not look as silly as it did before.

> The current VM handles things generally well imho, but has problems
> regulating itself under load.  My test load hits the VM right in it's
> weakest point (not _that_ weak, but..) by starting at zero and building
> rapidly to max.. and keeping it _right there_.
>
> > I'm not saying that it's a completely uninteresting case when your active
> > memory is bigger than you RAM of course, but perhaps there should be other
> > algorithms handling that case, such as putting some of the swapping
> > processes to sleep for some time, especially if you have lots of processes
> > competing for the memory. I may be wrong, but it seems to me that your
> > testcase falls into this second category (also known as thrashing).
>
> Thrashing?  Let's look some numbers. (not the ugly ones, the ~ok ones;)
>
> real    9m12.198s  make -j 30 bzImage
> user    7m41.290s
> sys     0m34.840s
> user  :       0:07:47.69  76.8%  page in :   452632
> nice  :       0:00:00.00   0.0%  page out:   399847
> system:       0:01:17.08  12.7%  swap in :    75338
> idle  :       0:01:03.97  10.5%  swap out:    88291
>
> real    8m6.994s  make bzImage
> user    7m34.350s
> sys     0m26.550s
> user  :       0:07:37.52  78.4%  page in :    90546
> nice  :       0:00:00.00   0.0%  page out:    18164
> system:       0:01:26.13  14.8%  swap in :        1
> idle  :       0:00:39.69   6.8%  swap out:        0
>
> ...look at cpu utilization.  One minute +tiny change to complete the
> large job vs the small (VM footprint) job.
>
> The box is not thrashing, it's working it's little silicon butt off.
> What I'm testing is the VM's ability to handle load without thrashing
> so badly that it loses throughput bigtime, stalls itself whatever..
> it's ability to regulate itself.  I consider a minute and a half to
> be ~acceptable, a minute to be good, and 30 seconds to be excellent.
> That's just my own little VM performance thermometer.

Why isn't user+system+idle == real?  SMP?

> > An at last, a humble request:  Every problem I've had with the VM has been
> > that it either swapped out too many processes and used too much cache, or
> > the other way around.  I'd really enjoy a way to tune this behaviour, if
> > possible.
>
> Tunables aren't really practical in VM (imho).  If there were a dozen
> knobs, you'd have to turn a dozen knobs a dozen times a day.  VM has
> to be self regulating.

Yes, that is of course the goal, but I'm suggesting that we would reach
the goal of a self-optimizing VM faster, if there were tunables to play
with.  The human brain is a very good optimizer.

> In case you can't tell (the length of this reply) I like my fovorite
> little generic throughput test a LOT :-)

Point taken.  :-)

/Tobias

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: VM Report was:Re: Break 2.4 VM in five easy steps

[Mike Galbraith]

On Fri, 8 Jun 2001, Tobias Ringstrom wrote:

> On Fri, 8 Jun 2001, Mike Galbraith wrote:
> > On Fri, 8 Jun 2001, Tobias Ringstrom wrote:
> > > On Fri, 8 Jun 2001, Mike Galbraith wrote:
> > > > I gave this a shot at my favorite vm beater test (make -j30 bzImage)
> > > > while testing some other stuff today.
> > >
> > > Could you please explain what is good about this test?  I understand that
> > > it will stress the VM, but will it do so in a realistic and relevant way?
> >
> > Can you explain what is bad about this test? ;)  It spins the same VM wheels
>
> I think a load of ~30 is quit uncommon, and therefor it is unclear to me
> that it would be a test that would be repesentative of most normal loads.

It's not supposed to be repesentative.  It's supposed to take the box
rapidly (but not instantly) from idle through lo->medium->high and
maintain solid throughput.

> > as any other load does.  What's the difference if I have a bunch of httpd
> > allocating or a bunch of cc1/as/ld?  This load has a modest cachable data
> > set and is compute bound.. and above all gives very repeatable results.
>
> Not a big difference.  The difference I was thinking abount is the
> difference between spawning lots of processes allocating, using and
> freeing lots of memory, compared to a case where you have a few processes
> touching a lot of already allocated pages in some pattern.  I was
> wondering whether optimizing for your case would be good or bad for the
> other case.  I know, I know, I should do more testing myself.  And I
> should probably not ask you, since you really really like your test,
> and you will probably just say yes... ;-)

It's not a matter of optimizing for my case.. that would be horrible.
It's a matter of is the vm capable of rapid and correct responses.

> At home, I'm running a couple of computers.  One of them is a slow
> computer running Linux, serving mail, NFS, SMB, etc.  I'm usually logged
> in on a couple of virtual consoles.  On this machine, I do not mind if all
> shells, daemons and other idle processes are beeing swapped out in favor
> of disk cache for the NFS and SMB serving.  In fact, that is a very good
> thing, and I want it that way.
>
> Another maching is my desktop machine.  When using this maching, I really
> hate when my emacsen, browsers, xterms, etc are swapped out just to give
> me some stupid disk cache for my xmms or compilations.  I do not care if a
> kernel compile is a little slower as long as my applications are snappy.
>
> How could Linux predict this?  It is a matter of taste, IMHO.

I have no idea.  It would be _wonderful_ if it could detect interactive
tasks and give them preferencial treatment.

> > I use it to watch reaction to surge.  I watch for the vm to build to a
> > solid maximum throughput without thrashing.  That's the portion of VM
> > that I'm interested in, so that's what I test.  Besides :) I simply don't
> > have the hardware to try to simulate hairy chested server loads.  There
> > are lots of folks with hairy chested boxes.. they should test that stuff.
>
> Agreed.  More testing is needed.  Now if we would have those knobs and
> wheels to turn, we could perhaps also tune our systems to behave as we
> like them, and submit that as well.  Right now you need to be a kernel
> hacker, and see through all the magic with shm, mmap, a bunch of caches,
> page lists, etc.  I'd give a lot for a nice picture (or state diagram)
> showing the lifetime of a page, but I have not found such a picture
> anywhere.  Besides, the VM seems to change every new release anyway.
>
> > I've been repeating ~this test since 2.0 times, and have noticed a 1:1
> > relationship.  When I notice that my box is ~happy doing this load test,
> > I also notice very few VM gripes hitting the list.
>
> Ok, but as you say, we need more tests.
>
> > > Isn't the interesting case when you have a number of processes using lots
> > > of memory, but only a part of all that memory is beeing actively used, and
> > > that memory fits in RAM.  In that case, the VM should make sure that the
> > > not used memory is swapped out.  In RAM you should have the used memory,
> > > but also disk cache if there is any RAM left.  Does the current VM handle
> > > this case fine yet?  IMHO, this is the case most people care about.  It is
> > > definately the case I care about, at least. :-)
> >
> > The interesting case is _every_ case.  Try seeing my particular test as
> > a simulation of a small classroom box with 30 students compiling their
> > assignments and it'll suddenly become quite realistic.  You'll notice
> > by the numbers I post that I was very careful to not overload the box in
> > a rediculous manner when selecting the total size of the job.. it's just
> > a heavily loaded box.  This test does not overload my IO resources, so
> > it tests the VM's ability to choose and move the right stuff at the right
> > time to get the job done with a minimum of additional overhead.
>
> I did not understand those numbers when I saw them the first time.  Now, I
> must say that your test does not look as silly as it did before.

[snip.. save a tree]

> Why isn't user+system+idle == real?  SMP?

Good question, no smp (sniff) here.

> > Tunables aren't really practical in VM (imho).  If there were a dozen
> > knobs, you'd have to turn a dozen knobs a dozen times a day.  VM has
> > to be self regulating.
>
> Yes, that is of course the goal, but I'm suggesting that we would reach
> the goal of a self-optimizing VM faster, if there were tunables to play
> with.  The human brain is a very good optimizer.

You bet!  The CPU is a stupid robot.  I've tried to think up good generic
tunables, and failed.  This is something that more folks should give some
thought.  Maybe someone will think of knobs that _are_ practical.

> > In case you can't tell (the length of this reply) I like my fovorite
> > little generic throughput test a LOT :-)
>
> Point taken.  :-)

	Cheers,

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/