Multiqueue VM Patch OK? Does page-aging really work?

Multiqueue VM Patch OK? Does page-aging really work?

[Benjamin Redelings I]

Hi guys - 
	I just want to remind people that dbench, bonnie, mmap2, and stuff are
only one type of way to measure the "performance" or the multiqueue
patch.  Even if these give good numbers (which they apparently don't
always), we still have major problems with the multiqueue patch, that I
think prevent it from being a candidate for 2.4.0 in its current state. 
Simply fixing the crashes is not good enough:

	Problem #1: 
	Simply untarring a large file (e.g. linux-2.4.0-test7.tar) evicts most
of the working set.  Example: I was running a few programs, including
netscape, with little swap in use.  Then I untarred a linux source tree,
and lo and behold, look at this!  44Mb swap!
telomere:~> cat free_after_untar 
             total       used       free     shared    buffers    
cached
Mem:         62872      61432       1440          0       1588     
41800
-/+ buffers/cache:      18044      44828
Swap:       128484      44044      84440

Now I have __44Mb__ of swap??  Interestingly there is about 41Mb cached.
One question I guess that needs to be asked is: is this cached data
"dirty data" in the newly created "linux/" tree , or is it "clean data"
from "linux.tar"?

Maybe the problem is that dirty data is given preference over other
data, and tyrannically takes over the cache. (But that couldn't be the
whole problem, see rest of this e-mail)

Anyway, for comparison, here is test8 after untarring a similar file
telomere:/usr/src/temp> free
             total       used       free     shared    buffers    
cached
Mem:         62944      60980       1964          0        752     
30900
-/+ buffers/cache:      29328      33616
Swap:       128484      13724     114760


	Problem #2:
	Programs that are basically unused, like (for example) an apache server
that I am running on my home computer, no longer have their RSS go down
to 4K.  Simply put, unused processes are not evicted, while data from
used processes IS evicted.

	Conclusion:
	1. Aren't BOTH these problems supposed to be solved by page aging?
Then, shouldn't the multiqueue-patched kernel do BETTER than test8? 
Apparently page-aging is not quite doing its job.  Why?
	2. With the drop_behind stuff, I'm sure the kernel will perform better,
at least with problem #1, to some extent.  However, even if the
drop_behind stuff is moved to the VMA level, I still think this is a
"special case hack".  I am not trying to be overly negative or critical
- it is just that the NEED for drop_behind this indicates that page
aging (the general solution) is not working.  Or am I missing something?

	In any case, what I am not missing is the fact that the multi-queue
patch is failing to reform the VM system in some of its most important
aspects.  I don't see how it could go into 2.4.0 in its current state.

keep up the good work!
-BenRI
-- 
"I want to be in the light, as He is in the Light,
 I want to shine like the stars in the heavens." - DC Talk, "In the
Light"
Benjamin Redelings I      <><     http://www.bol.ucla.edu/~bredelin/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Multiqueue VM Patch OK? Does page-aging really work?

[Rik van Riel]

On Fri, 8 Sep 2000, Benjamin Redelings I wrote:

> 	Problem #1: 
> 	Simply untarring a large file (e.g. linux-2.4.0-test7.tar) evicts most
> of the working set.  Example: I was running a few programs, including
> netscape, with little swap in use.  Then I untarred a linux source tree,
> and lo and behold, look at this!  44Mb swap!
> 
> Now I have __44Mb__ of swap??  Interestingly there is about 41Mb
> cached. One question I guess that needs to be asked is: is this
> cached data "dirty data" in the newly created "linux/" tree , or
> is it "clean data" from "linux.tar"?

Indeed, this is something which still has to be fixed.
You have to keep in mind that the new VM has received
about 5 days of performance tweaking right now and has
only received very limited testing. Most of the time
working on the new VM has been spent tracking down the
(now fixed) SMP bug...

> Maybe the problem is that dirty data is given preference over
> other data, and tyrannically takes over the cache.

Please try to understand the code before saying something
like this.

We age all active pages the same and the distiction is only
made between pages which are already "evicted" from the
active queue and moved to an inactive queue, so aging and
flushing definately do NOT interfere with each other.

> 	Problem #2:
> 	Programs that are basically unused, like (for example) an
> apache server that I am running on my home computer, no longer
> have their RSS go down to 4K.  Simply put, unused processes are
> not evicted, while data from used processes IS evicted.

This is not a problem but only looks like one. We only evict
pages which are not in active use anywhere, but keep shared
pages from glibc in memory (and mapped) when they are in
active use in other processes.

> 	Conclusion:
> 	1. Aren't BOTH these problems supposed to be solved by page aging?
> Then, shouldn't the multiqueue-patched kernel do BETTER than test8? 
> Apparently page-aging is not quite doing its job.  Why?

The one problem you saw is indeed in need of some tuning
and I'm working on it right now. One of the things which
would help identifying all the problems (so we can get
them fixed faster) would be having a larger test base..

> 	2. With the drop_behind stuff, I'm sure the kernel will perform better,
> at least with problem #1, to some extent.  However, even if the
> drop_behind stuff is moved to the VMA level, I still think this is a
> "special case hack".  I am not trying to be overly negative or critical
> - it is just that the NEED for drop_behind this indicates that page
> aging (the general solution) is not working.  Or am I missing something?

While page aging can be used to make the distinction between
pages which have been used once and pages which are being
used all the time, every use-once page puts a little bit of
"pressure" on the working set and it is still possible to
completely overwhelm (and evict) the working set by doing
very fast streaming IO.

Drop-behind is a way to detect the difference between pages
which are used once and pages which are used more often, while
at the same time not putting the pages we just read ahead in
jeopardy of being evicted before they are being put to use.

> 	In any case, what I am not missing is the fact that the
> multi-queue patch is failing to reform the VM system in some of
> its most important aspects.  I don't see how it could go into
> 2.4.0 in its current state.

Define "most important". You only gave me one thing
which still needs to be tweaked a bit and an optical
illusion with no memory use in reality.

Things like interactive latency and speed increase
for workloads which exhibit locality of reference
(say, big compiles) are important as well...

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Happiness with t8-vmpatch4 (was Re: Does page-aging really work?)

[Benjamin Redelings I]

Hi guys - 
	I have done a few small tests on t8-vmpatch4 (64Mb RAM, 166Mhz UP
PPro), and I must say that I am very happy with the results.  More
comparison with test8 vanilla confirms that Rik's version is much more
responsive, also that Rik is (or course) correct - unused process are
indeed swapped out, but shared pages from (I assume) libc remain.

	Observations:
   1. test8-vmpatch4 does not swap very much at first, but then swaps a
lot of memory in a short time when triggered.
	Specifically, I untarred a kernel source tree and saw NO swapping . 
The working set was NOT evicted :)  I then started more programs, and
saw that the cache shrunk a lot: still no swapping.  Then I ran the
untar again, and the kernel swapped out 22Mb very quickly.  However, I
must say that it was still very smooth - I didn't even notice the
swapping until I looked at xosview; normally it is quite audible.  Also,
the choice of pages for swapping out seemed to be VERY accurate - there
was virtually no page-in as long as I was watching...

  2. I guess we could wish that unused programs got swapped a bit sooner
instead of all at once - but presumably that can be tuned.

	Congrats on the great code!  Tuning does indeed seem to be fixing the
problems that I saw before.

thanks again,
-BenRI
-- 
"I want to be in the light, as He is in the Light,
 I want to shine like the stars in the heavens." - DC Talk, "In the
Light"
Benjamin Redelings I      <><     http://www.bol.ucla.edu/~bredelin/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Happiness with t8-vmpatch4 (was Re: Does page-aging really work?)

[Rik van Riel]

On Sat, 16 Sep 2000, Benjamin Redelings I wrote:

>    1. test8-vmpatch4 does not swap very much at first, but then
> swaps a lot of memory in a short time when triggered.

>   2. I guess we could wish that unused programs got swapped a
> bit sooner instead of all at once - but presumably that can be
> tuned.

This is indeed something to look at. Maybe we could give
idle processes (sleeping for more than 20 seconds?) a
"full" swap_cnt instead of swap_cnt = rss >> SWAP_SHIFT ?

And we could also start swapping a bit earlier when the
cache is getting small, but I'm not sure about how to
do this or exactly what performance benefits that would
give ...

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Happiness with t8-vmpatch4 (was Re: Does page-aging really work?)

[Benjamin Redelings I]

> This is indeed something to look at. Maybe we could give
> idle processes (sleeping for more than 20 seconds?) a
> "full" swap_cnt instead of swap_cnt = rss >> SWAP_SHIFT ?
 
This doesn't seem like it should be necessary.  Right now, unused
processes ARE swapped preferentially (and completely) - its just that
swapping happens all of a sudden.

> And we could also start swapping a bit earlier when the
> cache is getting small, but I'm not sure about how to
> do this or exactly what performance benefits that would
> give ...

	Evicting unused pages, either from the cache or from process  can have
significant benefits on my machine (64Mb).  Once swapping triggered,
20Mb were paged out, and stayed out.  If these 20 Mb had been paged out
before, then I would have had 20Mb more cache to work with, which is 31%
of my memory.  Go figure :)
	While I agree with Byron that on low memory machines a smaller cache
can be a good thing - this DOES depend on the amount of RAM, and on the
workload.  But on higher memory machines, more aggressiveness against
program code is good.  Is there a way to make this adjust itself
dynamically - e.g. by measuring page faults?
	Anyway - I have seen the behavior that Byron described, where 'used'
increases as a result of a 'find'.  I think that maybe some more
aggressiveness against code pages (is that the right phrase?) might
solve the swapping problem and improve performance on low memory
machines also (since some code pages are simply evicted).

BTW, with test8-vmpatch4, I am gettings zillions of "VM: page_launder,
found pre-cleaned page ?!" messages.

-BenRI
-- 
"I want to be in the light, as He is in the Light,
 I want to shine like the stars in the heavens." - DC Talk, "In the
Light"
Benjamin Redelings I      <><     http://www.bol.ucla.edu/~bredelin/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Happiness with t8-vmpatch4 (was Re: Does page-aging really work?)

[Rik van Riel]

On Sat, 16 Sep 2000, Benjamin Redelings I wrote:

> > This is indeed something to look at. Maybe we could give
> > idle processes (sleeping for more than 20 seconds?) a
> > "full" swap_cnt instead of swap_cnt = rss >> SWAP_SHIFT ?
>
>  This doesn't seem like it should be necessary.  Right now,
> unused processes ARE swapped preferentially (and completely) -
> its just that swapping happens all of a sudden.

We may be able to fix that by swapping long-idle processes
before we have eaten all of the cache. 

> 	Evicting unused pages, either from the cache or from
> process can have significant benefits on my machine (64Mb).  
> Once swapping triggered, 20Mb were paged out, and stayed out.  
> If these 20 Mb had been paged out before, then I would have had
> 20Mb more cache to work with, which is 31% of my memory.  Go
> figure :)

*nod*

> BTW, with test8-vmpatch4, I am gettings zillions of "VM:
> page_launder, found pre-cleaned page ?!" messages.

No you're not. That's test8-vmpatch3.
This is fixed in -vmpatch4...

(and the comments in the code have been
updated to explain how and what)

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/