shrink_mmap() change in ac-21

shrink_mmap() change in ac-21

[Zlatko Calusic]

Hi, Alan and others!

The shrink_mmap() change in your latest prepatch (ac12) doesn't look
very healthy. Removing the test for the wrong zone we effectively
discard lots of wrong pages before we get to the right one. That is
effectively flushing the page cache and we have unbalanced system.

For example, check the "vmstat 1" output below, done while I was
reading a big file from the disk. At some point in time, the page
cache shrunk to almost half of its size (75MB -> 42MB).

The reason is balancing of the DMA zone (which is much smaller on a
128MB machine than the NORMAL zone!). shrink_mmap() now happily evicts
wrong pages from the memory and continues doing so until it finally
frees enough pages from the DMA zone. That, of course, hurts caching
as the page cache gets shrunk a lot without a good reason.


   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  1  0    988   5556    172  78708   0   0  3811     0  342   859   0   6  93
 0  1  0   1040   9712    176  74640   0 168  3043    42  309   801   0   6  93
 0  1  0   1084   7272    184  76800   0 408  2659   102  317   762   0   7  93
 0  1  0   1084   8704__  212  75308   0   0  2730     0  285   782   0   7  93
 3  0  0   1084  42400  \ 192  42780   0   0  2447     0  270   703   0   8  91
 0  1  0   1084  30092  | 204  54684   0   0  2979     0  299   767   1   4  95
                         \
                           here! :)

The incriminating change is:


Index: 24001.28/mm/filemap.c
--- 24001.28/mm/filemap.c Wed, 14 Jun 2000 01:44:09 +0200 zcalusic (linux/F/b/16_filemap.c 1.6.1.3.2.4.1.1.2.2.2.1.1.21.1.1.3.2.3.1.4.1 644)
+++ 24001.31(w)/mm/filemap.c Sun, 18 Jun 2000 21:23:47 +0200 zcalusic (linux/F/b/16_filemap.c 1.6.1.3.2.4.1.1.2.2.2.1.1.21.1.1.3.2.3.1.4.2 644)
@@ -361,16 +361,6 @@
 			}
 		}
 
-		/*
-		 * Page is from a zone we don't care about.
-		 * Don't drop page cache entries in vain.
-		 */
-		if (page->zone->free_pages > page->zone->pages_high) {
-			/* the page from the wrong zone doesn't count */
-			count++;
-			goto unlock_continue;
-		}
-
 		/* Take the pagecache_lock spinlock held to avoid
 		   other tasks to notice the page while we are looking at its
 		   page count. If it's a pagecache-page we'll free it

Regards,
-- 
Zlatko
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Rik van Riel]

On 19 Jun 2000, Zlatko Calusic wrote:

> The shrink_mmap() change in your latest prepatch (ac12) doesn't look
> very healthy. Removing the test for the wrong zone we effectively
> discard lots of wrong pages before we get to the right one. That is
> effectively flushing the page cache and we have unbalanced system.
> 
> For example, check the "vmstat 1" output below, done while I was
> reading a big file from the disk. At some point in time, the page
> cache shrunk to almost half of its size (75MB -> 42MB).
> 
> The reason is balancing of the DMA zone (which is much smaller on a
> 128MB machine than the NORMAL zone!). shrink_mmap() now happily evicts
> wrong pages from the memory and continues doing so until it finally
> frees enough pages from the DMA zone. That, of course, hurts caching
> as the page cache gets shrunk a lot without a good reason.

I already suspected this could happen.

/me looks at quintela

Juan, didn't you and Roger have a patch to solve this? ;)
(I think quintela and roger already have a patch so I'll save
writing one myself)

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

Wanna talk about the kernel?  irc.openprojects.net / #kernelnewbies
http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Jamie Lokier]

Zlatko Calusic wrote:
> The shrink_mmap() change in your latest prepatch (ac12) doesn't look
> very healthy. Removing the test for the wrong zone we effectively
> discard lots of wrong pages before we get to the right one. That is
> effectively flushing the page cache and we have unbalanced system.

You know, there may be some sense in removing pages from the wrong zone,
if those wrong zones are quite full.  If the DMA zone desparately needs
free pages and keeps needing them, isn't it good to encourage future
non-DMA allocations to use another zone?  Removing pages from other
zones is one way to achieve that.

-- Jamie
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Manfred Spraul, Zlatko Calusic]

>
> The reason is balancing of the DMA zone (which is much smaller on a
> 128MB machine than the NORMAL zone!). shrink_mmap() now happily evicts
> wrong pages from the memory and continues doing so until it finally
> frees enough pages from the DMA zone. That, of course, hurts caching
> as the page cache gets shrunk a lot without a good reason.
>
What caused the zone balancing?
Did you deliberately allocate GFP_DMA memory (sound card, old scsi card,
floppy disk, ...) or was it during "normal" operation?

--
    Manfred

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Rik van Riel]

On Mon, 19 Jun 2000, Jamie Lokier wrote:
> Zlatko Calusic wrote:
> > The shrink_mmap() change in your latest prepatch (ac12) doesn't look
> > very healthy. Removing the test for the wrong zone we effectively
> > discard lots of wrong pages before we get to the right one. That is
> > effectively flushing the page cache and we have unbalanced system.
> 
> You know, there may be some sense in removing pages from the
> wrong zone, if those wrong zones are quite full.

If the zone is full, it can't be a "wrong zone". The problem
was that we kept removing pages from zones they shouldn't be
removed from. If a zone has zone->free_pages > zone->pages_high,
we should stop freeing pages from that zone.

> If the DMA zone desparately needs free pages and keeps needing
> them, isn't it good to encourage future non-DMA allocations to
> use another zone?

Ahh, but we already do this (up to zone->pages_high). It just
doesn't make sense to keep doing this infinitely ;)

Please wait a few more minutes for a patch which should fix it.
I've assembled a very conservative patch set, grabbing bits from
patches by Juan Quintela, Roger Larson and one minute snippet
from the old 2.3 code...

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

Wanna talk about the kernel?  irc.openprojects.net / #kernelnewbies
http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Andrea Arcangeli]

On Mon, 19 Jun 2000, Rik van Riel wrote:

>Ahh, but we already do this (up to zone->pages_high). It just

more precisely up to zone->pages_high - zone->pages_low/min.

Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Andrea Arcangeli]

On Mon, 19 Jun 2000, Jamie Lokier wrote:

>if those wrong zones are quite full.  If the DMA zone desparately needs
>free pages and keeps needing them, isn't it good to encourage future
>non-DMA allocations to use another zone?  Removing pages from other

After some time the DMA zone will be full again anyway and you payed a
cost that consists in throwing away unrelated innocent pages. I'm not
convinced it's the right thing to do.

Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Zlatko Calusic]

"Manfred Spraul" <manfred@colorfullife.com> writes:

> From: "Zlatko Calusic" <zlatko@iskon.hr>
> >
> > The reason is balancing of the DMA zone (which is much smaller on a
> > 128MB machine than the NORMAL zone!). shrink_mmap() now happily evicts
> > wrong pages from the memory and continues doing so until it finally
> > frees enough pages from the DMA zone. That, of course, hurts caching
> > as the page cache gets shrunk a lot without a good reason.
> >
> What caused the zone balancing?
> Did you deliberately allocate GFP_DMA memory (sound card, old scsi card,
> floppy disk, ...) or was it during "normal" operation?
> 

No, I haven't done anything special with the DMA zone. But pages get
allocated from the DMA zone normally (it is almost 16MB of free RAM,
after all).

Then when kswapd kicks in because free memory in the DMA zone got low,
it starts freeing pages until we free enough pages from the DMA
zone. But it doesn't check if such a freeing hurts other zones.

Simple mathematics: On a 128MB machine, DMA zone is 16MB, thus NORMAL
zone is 112MB. 112/16 = 7. So statistically, for every DMA page freed,
we free another SEVEN! pages from the NORMAL zone. And we won't stop
doing such a genocide until DMA zone recovers.

That was 128MB machine, consider how the problem gets progressively
worse on machines with more RAM.
-- 
Zlatko
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Zlatko Calusic]

Andrea Arcangeli <andrea@suse.de> writes:

> On Mon, 19 Jun 2000, Jamie Lokier wrote:
> 
> >if those wrong zones are quite full.  If the DMA zone desparately needs
> >free pages and keeps needing them, isn't it good to encourage future
> >non-DMA allocations to use another zone?  Removing pages from other
> 
> After some time the DMA zone will be full again anyway and you payed a
> cost that consists in throwing away unrelated innocent pages. I'm not
> convinced it's the right thing to do.
> 

Exactly!
-- 
Zlatko
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Manfred Spraul, Zlatko Calusic]

>
> Simple mathematics: On a 128MB machine, DMA zone is 16MB, thus NORMAL
> zone is 112MB. 112/16 = 7. So statistically, for every DMA page freed,
> we free another SEVEN! pages from the NORMAL zone. And we won't stop
> doing such a genocide until DMA zone recovers.
>
I'm also concerned about 1GB boxes:
the highmem zone only contains ~ 64 MB (or 128?), and so most allocations go
into a tiny zone and are then "downgraded" to GFP_NORMAL.

Perhaps we should switch to per-zone lru lists?

--
    Manfred

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Rik van Riel]

On Tue, 20 Jun 2000, Andrea Arcangeli wrote:
> On Mon, 19 Jun 2000, Jamie Lokier wrote:
> 
> >if those wrong zones are quite full.  If the DMA zone desparately needs
> >free pages and keeps needing them, isn't it good to encourage future
> >non-DMA allocations to use another zone?  Removing pages from other
> 
> After some time the DMA zone will be full again anyway and you
> payed a cost that consists in throwing away unrelated innocent
> pages. I'm not convinced it's the right thing to do.

I didn't know for sure either until I tested -ac21 on my
192MB workstation. The bursts kswapd went through when
it was freeing DMA memory (and 8MB of other memory) have
convinced me that this is not a good idea.

Also, since kswapd stops when all zones have free_pages
above pages_low and we'll free up to pages_high pages of
one zone, it means that we'll:

- allocate the next series of pages from that one zone
  with tons of unused pages
- wake up kswapd so we'll free the *next* unused pages
  from that zone when we run out of the current batch
- rinse and repeat

This means we'll do a *lot* more allocations from the
less loaded zones than from the other zone, with a few
(short) interruptions by kswapd. Also, there's no need
to throw away data early.

Of course, once we have a scavenge list (in the active
inactive scavenge list VM) this whole point will be moot
and we just want to avoid doing too much IO at once).

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

Wanna talk about the kernel?  irc.openprojects.net / #kernelnewbies
http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Juan J. Quintela]

>>>>> "rik" == Rik van Riel <riel@conectiva.com.br> writes:

Hi

rik> Also, since kswapd stops when all zones have free_pages
rik> above pages_low and we'll free up to pages_high pages of
rik> one zone, it means that we'll:

rik> - allocate the next series of pages from that one zone
rik>   with tons of unused pages
rik> - wake up kswapd so we'll free the *next* unused pages
rik>   from that zone when we run out of the current batch
rik> - rinse and repeat

That is what my change to page_alloc does, it makes more probable to
get pages from zones that have more than page_high free pages.  That
means that it is less probable to get one zone with less than page_low
free pages and other with a lot of free pages.

Notice that this behaviour happens also in my box where there is no
ISA cards at all, and I have to wait for a page to become free in the
DMA zone.  Is there some way to need a DMA page in a machine without
any ISA card?  If not, it could be a good Idea to have only one zone
in machines that haven't ISA cards and have less than 1GB of RAM.

Later, Juan.

-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[willy]

On Tue, Jun 20, 2000 at 06:14:33PM +0200, Manfred Spraul wrote:
> I'm also concerned about 1GB boxes:
> the highmem zone only contains ~ 64 MB (or 128?), and so most allocations go
> into a tiny zone and are then "downgraded" to GFP_NORMAL.

Not that I want to get involved with the VM system in _any way at all_,
but bcrl pointed out that highmem doesn't really cost a lot, so why not
change to:

3GB user space
512MB vmalloc space
512MB kernel space

i don't think there are many machines with amounts of RAM between 512MB
and 768MB so the worst case is that only 1/3 of the RAM is high, instead
of 1/8 (vmalloc is currently 128MB of 1GB).  i know jeff garzik wants
more vmalloc space for a frame grabber so maybe now is the right time
to make that change?

-- 
The Sex Pistols were revolutionaries.  The Bay City Rollers weren't.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Alan Cox]

> Not that I want to get involved with the VM system in _any way at all_,
> but bcrl pointed out that highmem doesn't really cost a lot, so why not
> change to:

A lot of stuff has to be in the low block. It would hurt 8Gig users badly
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Manfred Spraul, Juan J. Quintela]

>
> Notice that this behaviour happens also in my box where there is no
> ISA cards at all, and I have to wait for a page to become free in the
> DMA zone.  Is there some way to need a DMA page in a machine without
> any ISA card?  If not, it could be a good Idea to have only one zone
> in machines that haven't ISA cards and have less than 1GB of RAM.
>
How do you want to find out that a box has no ISA card?
Additionally, the floppy disk needs GFP_DMA memory and IIRC some non-ISA
sound cards have < 32 (28?) address lines.

--
    Manfred



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Juan J. Quintela]

>>>>> "manfred" == Manfred Spraul <manfred@colorfullife.com> writes:

manfred> From: "Juan J. Quintela" <quintela@fi.udc.es>
>> 
>> Notice that this behaviour happens also in my box where there is no
>> ISA cards at all, and I have to wait for a page to become free in the
>> DMA zone.  Is there some way to need a DMA page in a machine without
>> any ISA card?  If not, it could be a good Idea to have only one zone
>> in machines that haven't ISA cards and have less than 1GB of RAM.
>> 
manfred> How do you want to find out that a box has no ISA card?
manfred> Additionally, the floppy disk needs GFP_DMA memory and IIRC some non-ISA
manfred> sound cards have < 32 (28?) address lines.

I have no idea how to find out that a BOX have no ISA cards.  That was
only a suggestion.  Perhaps asking if you want ISA zone during
configuration??  I have no idea about floppy and some non ISA cards,
they are a problem.  But the point here is that I am not using floppy
on that machine, no Sound Card on that machine, and I have to wait for
a DMA page to become free (when nobody is asking, will ask for DMA
memory...)  Perhaps it is too late to solve that problem for 2.4, but
it appears that somebody needs to think a bit about the problem.

Later, Juan.

-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Andrea Arcangeli]

On 20 Jun 2000, Juan J. Quintela wrote:

>memory...)  Perhaps it is too late to solve that problem for 2.4, but
>it appears that somebody needs to think a bit about the problem.

Incidentally I just thought about this and I fixed the problem at
2.3.99-pre2 time. With classzone design if nobody is doing a GFP_DMA
allocation (and nobody is doing that because as you said you don't have
soundcard and you don't use the floppy) then _nobody_ will ever to try to
take some page free from the DMA zone. The DMA zone can be shrink of
course but it will be considered as a whole with the NORMAL zone. Or see
it in another way: when you'll do a GFP_KERNEL allocation the kernel will
behave exactly if you would have only one zone (if you have less than 1
giga of ram of course).

Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: shrink_mmap() change in ac-21

[Rik van Riel]

On Tue, 20 Jun 2000, Linus Torvalds wrote:
> In article <Pine.LNX.4.21.0006201258190.12944-100000@duckman.distro.conectiva>,
> Rik van Riel  <riel@conectiva.com.br> wrote:
> >
> >I didn't know for sure either until I tested -ac21 on my
> >192MB workstation. The bursts kswapd went through when
> >it was freeing DMA memory (and 8MB of other memory) have
> >convinced me that this is not a good idea.
> 
> Note that this is exactly what the zone "goodness" test is
> supposed to avoid.

Indeed, and reinserting the zone goodness test makes the system
perform wonderfully again. It was removed shortly in -ac21 for
unknown reasons.  Removing that test was done by a few people on
IRC when we tried to identify if that was the cause of high
kswapd cpu use on low-memory machines (do_try_to_free_pages would
call shrink_mmap until count reaches 0) and it was never intended
to go into the kernel...

> I suspect that the problem is that the goodness test is wrong.

No, the current zone goodness test is very much ok.

> 	if (page->zone->zone_wake_kswapd)
> 		/* uninteresting page */
> 
> rather than the current test
> 
> 	if (page->zone->free_pages > page->zone->pages_high)
> 		/* uninteresting page */
> 
> Note that once the zone has actually been low on memory, the two
> tests are equivalent:

This is exactly what we need to avoid. If one zone is slightly
loaded and another zone contains a ton of old pages, we want to
free the old pages from the other zone regardless, until we reach
pages_high.

Suppose, as an example, that the normal zone contains a ton of
old pages (but it has not yet reached the low watermark) and we
do a DMA allocation. The DMA allocation wakes up kswapd and the
system tries to free some pages.

In this case we *really* want to free some of the (stone age old)
pages from the normal zone so we'll allocate more pages from that
zone (and, relatively, less pages from the dma zone) until we
wake up kswapd again.

In fact, the high and low watermark (kswapd woken up and going
to sleep) are pretty much the same, modulo time difference. The
hysteresis in the latest kernel is achieved by two effects. One
of them is the kswapd_pause logic in __alloc_pages and the other
is the fact that often the oldest pages in one zone have a
different age (and thus position) than the pages from the other
zone(s). This makes us free "extra" pages from the less loaded
zone and will balance the load between the zones faster (I hope).

> The zone test is _definitely_ needed, because without that test
> we'll deplete zones that have tons of memory and really should
> not be depleted..

*nod*

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

Wanna talk about the kernel?  irc.openprojects.net / #kernelnewbies
http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/