[PATCH] vma limited swapin readahead

[PATCH] vma limited swapin readahead

[Marcelo Tosatti]

Hi, 

The current swapin readahead code reads a number of pages (1 >>
page_cluster)  which are physically contiguous on disk with reference to
the page which needs to be faulted in.

However, the pages which are contiguous on swap are not necessarily
contiguous in the virtual memory area where the fault happened. That means
the swapin readahead code may read pages which are not related to the
process which suffered a page fault.

I've changed the swapin code to not readahead pages if they are not
virtually contiguous on the vma which is being faulted to avoid
the problem described above.

Testers are very welcome since I'm unable to test this in various
workloads.

The patch is available at
http://bazar.conectiva.com.br/~marcelo/patches/v2.4/2.4.1pre10/swapin_readahead.patch


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Stephen C. Tweedie]

Hi,

On Wed, Jan 31, 2001 at 01:05:02AM -0200, Marcelo Tosatti wrote:
> 
> However, the pages which are contiguous on swap are not necessarily
> contiguous in the virtual memory area where the fault happened. That means
> the swapin readahead code may read pages which are not related to the
> process which suffered a page fault.
> 
Yes, but reading extra sectors is cheap, and throwing the pages out of
memory again if they turn out not to be needed is also cheap.  The
on-disk swapped pages are likely to have been swapped out at roughly
the same time, which is at least a modest indicator of being of the
same age and likely to have been in use at the same time in the past.

I'd like to see at lest some basic performance numbers on this,
though.

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Marcelo Tosatti]

On Wed, 31 Jan 2001, Stephen C. Tweedie wrote:

> Hi,
> 
> On Wed, Jan 31, 2001 at 01:05:02AM -0200, Marcelo Tosatti wrote:
> > 
> > However, the pages which are contiguous on swap are not necessarily
> > contiguous in the virtual memory area where the fault happened. That means
> > the swapin readahead code may read pages which are not related to the
> > process which suffered a page fault.
> > 
> Yes, but reading extra sectors is cheap, and throwing the pages out of
> memory again if they turn out not to be needed is also cheap.  The
> on-disk swapped pages are likely to have been swapped out at roughly
> the same time, which is at least a modest indicator of being of the
> same age and likely to have been in use at the same time in the past.

You're throwing away pages from memory to do the readahead. 

This pages might be more useful than the pages which you're reading from
swap. 

> I'd like to see at lest some basic performance numbers on this,
> though.

I'm not sure if limiting the readahead the way my patch does is a better
choice, too.

I posted it to lkml so people can test it under different workloads and
report results.

Thanks for your feedback. 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Eric W. Biederman]

Marcelo Tosatti <marcelo@conectiva.com.br> writes:

> On Wed, 31 Jan 2001, Stephen C. Tweedie wrote:
> 
> > Hi,
> > 
> > On Wed, Jan 31, 2001 at 01:05:02AM -0200, Marcelo Tosatti wrote:
> > > 
> > > However, the pages which are contiguous on swap are not necessarily
> > > contiguous in the virtual memory area where the fault happened. That means
> > > the swapin readahead code may read pages which are not related to the
> > > process which suffered a page fault.
> > > 
> > Yes, but reading extra sectors is cheap, and throwing the pages out of
> > memory again if they turn out not to be needed is also cheap.  The
> > on-disk swapped pages are likely to have been swapped out at roughly
> > the same time, which is at least a modest indicator of being of the
> > same age and likely to have been in use at the same time in the past.
> 
> You're throwing away pages from memory to do the readahead. 
> 
> This pages might be more useful than the pages which you're reading from
> swap. 

Possibly.  However the win (lower latency) from getting swapin
readahead is probably even bigger.  And you are throwing out the least
desirable pages in memory.

> > I'd like to see at lest some basic performance numbers on this,
> > though.
> 
> I'm not sure if limiting the readahead the way my patch does is a better
> choice, too.

A better choice is probably to make certain the read and write paths are in
sync so that you can know the readahead is going to do you some good.
This is a little tricky though.  

Unless you can see a big performance win somewhere please don't submit
this to Linus for inclusion.


Eric
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[David Gould]

On Wed, Jan 31, 2001 at 12:40:52PM -0700, Eric W. Biederman wrote:
> Marcelo Tosatti <marcelo@conectiva.com.br> writes:
> > On Wed, 31 Jan 2001, Stephen C. Tweedie wrote:
> > > On Wed, Jan 31, 2001 at 01:05:02AM -0200, Marcelo Tosatti wrote:
> > > > 
> > > > However, the pages which are contiguous on swap are not necessarily
> > > > contiguous in the virtual memory area where the fault happened. That means
> > > > the swapin readahead code may read pages which are not related to the
> > > > process which suffered a page fault.
> > > > 
> > > Yes, but reading extra sectors is cheap, and throwing the pages out of
> > > memory again if they turn out not to be needed is also cheap.  The
> > > on-disk swapped pages are likely to have been swapped out at roughly
> > > the same time, which is at least a modest indicator of being of the
> > > same age and likely to have been in use at the same time in the past.
> > 
> > You're throwing away pages from memory to do the readahead. 
> > 
> > This pages might be more useful than the pages which you're reading from
> > swap. 
> 
> Possibly.  However the win (lower latency) from getting swapin
> readahead is probably even bigger.  And you are throwing out the least
> desirable pages in memory.
> 
> > > I'd like to see at lest some basic performance numbers on this,
> > > though.
> > 
> > I'm not sure if limiting the readahead the way my patch does is a better
> > choice, too.
... 
> Unless you can see a big performance win somewhere please don't submit
> this to Linus for inclusion.

Hmmm, arguably reading pages we do not want is a mistake. I should think that
if a big performance win is required to justify a design choice, it should
be especially required to show such a win for doing something that on its
face is wrong.

I am skeptical of the argument that we can win by replacing "the least
desirable" pages with pages were even less desireable and that we have
no recent indication of any need for. It seems possible under heavy swap
to discard quite a portion of the useful pages in favor of junk that just
happenned to have a lucky disk address.

-dg

-- 
David Gould                                                 dg@suse.com
SuSE, Inc.,  580 2cd St. #210,  Oakland, CA 94607          510.628.3380
You left them alone in a room with a penguin?!  Mr Gates, your men are
already dead.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Eric W. Biederman]

David Gould <dg@suse.com> writes:

> Hmmm, arguably reading pages we do not want is a mistake. I should think that
> if a big performance win is required to justify a design choice, it should
> be especially required to show such a win for doing something that on its
> face is wrong.

The case for files and has already been justified.   
The performance gain of reading pages that are contiguous on disk has
been justified. 
The only problem thing that has not been shown is that swap pages that
are used together are located near each other in swap.

As for design choices simplicity, maintainability and
comprehensiblility, tend to be more important than absolute
performance.  This lets bugs be fixed, and the big changes that tend
to be the biggest wins happen.

> I am skeptical of the argument that we can win by replacing "the least
> desirable" pages with pages were even less desireable and that we have
> no recent indication of any need for. It seems possible under heavy swap
> to discard quite a portion of the useful pages in favor of junk that just
> happenned to have a lucky disk address.

I won't argue that.  My gut just says we should work to improve the
disk addresses, so it isn't luck. ;)  And only if we fail in that
hack up the efficient simple policy, that we have for reading disk
data in.

Of course since I'm not actually writing the code at the moment
this is all hot air :)

Eric

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Stephen C. Tweedie]

Hi,

On Wed, Jan 31, 2001 at 04:24:24PM -0800, David Gould wrote:
> 
> I am skeptical of the argument that we can win by replacing "the least
> desirable" pages with pages were even less desireable and that we have
> no recent indication of any need for. It seems possible under heavy swap
> to discard quite a portion of the useful pages in favor of junk that just
> happenned to have a lucky disk address.

When readin clustering was added to 2.2 for swap and paging,
performance for a lot of VM-intensive tasks more than doubled.  Disk
seeks are _expensive_.  If you read in 15 neighbouring pages on swapin
and on average only one of them turns out to be useful, you have still
halved the number of swapin IOs required.  The performance advantages
are so enormous that easily compensate for the cost of holding the
other, unneeded pages in memory for a while.

Also remember that the readahead pages won't actually get mapped into
memory, so they can be recycled easily.  So, under swapping you tend
to find that the extra readin pages are going to be replacing old,
unneeded readahead pages to some extent, rather than swapping out
useful pages.

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Marcelo Tosatti]

On Thu, 1 Feb 2001, Stephen C. Tweedie wrote:

> Hi,
> 
> On Wed, Jan 31, 2001 at 04:24:24PM -0800, David Gould wrote:
> > 
> > I am skeptical of the argument that we can win by replacing "the least
> > desirable" pages with pages were even less desireable and that we have
> > no recent indication of any need for. It seems possible under heavy swap
> > to discard quite a portion of the useful pages in favor of junk that just
> > happenned to have a lucky disk address.
> 
> When readin clustering was added to 2.2 for swap and paging,
> performance for a lot of VM-intensive tasks more than doubled.  Disk
> seeks are _expensive_.  If you read in 15 neighbouring pages on swapin
> and on average only one of them turns out to be useful, you have still
> halved the number of swapin IOs required.  The performance advantages
> are so enormous that easily compensate for the cost of holding the
> other, unneeded pages in memory for a while.
> 
> Also remember that the readahead pages won't actually get mapped into
> memory, so they can be recycled easily.  So, under swapping you tend
> to find that the extra readin pages are going to be replacing old,
> unneeded readahead pages to some extent, rather than swapping out
> useful pages.

If we're under free memory shortage, "unlucky" readaheads will be harmful.

Currently the swapin readahead code can block waiting for memory to do the
readahead, forcing other pages to be aged/freed more aggressively.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Stephen C. Tweedie]

Hi,

On Thu, Feb 01, 2001 at 08:53:33AM -0200, Marcelo Tosatti wrote:
> 
> On Thu, 1 Feb 2001, Stephen C. Tweedie wrote:
> 
> If we're under free memory shortage, "unlucky" readaheads will be harmful.

I know, it's a balancing act.  But given that even one successful
readahead per read will halve the number of swapin seeks, the
performance loss due to the extra scavenging has got to be bad to
outweigh the benefit.

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Rik van Riel]

On Thu, 1 Feb 2001, Stephen C. Tweedie wrote:
> On Thu, Feb 01, 2001 at 08:53:33AM -0200, Marcelo Tosatti wrote:
> > On Thu, 1 Feb 2001, Stephen C. Tweedie wrote:
> > 
> > If we're under free memory shortage, "unlucky" readaheads will be harmful.
> 
> I know, it's a balancing act.  But given that even one
> successful readahead per read will halve the number of swapin
> seeks, the performance loss due to the extra scavenging has got
> to be bad to outweigh the benefit.

But only when the extra pages we're reading in don't
displace useful data from memory, making us fault in
those other pages ... causing us to go to the disk
again and do more readahead, which could potentially
displace even more pages, etc...

One solution could be to put (most of) the swapin readahead
pages on the inactive_dirty list, so pressure by readahead
on the resident pages is smaller and the not used readahead
pages are reclaimed faster.

(and with the size of the inactive list being 1 second worth
of page steals, those pages still have a good chance of being
used before they're being recycled)

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Ingo Oeser]

On Thu, Feb 01, 2001 at 02:45:04PM -0200, Rik van Riel wrote:
> One solution could be to put (most of) the swapin readahead
> pages on the inactive_dirty list, so pressure by readahead
> on the resident pages is smaller and the not used readahead
> pages are reclaimed faster.

Shouldn't they be on inactive_clean anyway? They are not mapped
(if I read Stephens comment correctly) and are clean (because we
just read them in).

So if we have to put it there explicitly, we have at least a
performance bug, don't we?

Or do I still not get the new linux mm design? ;-(

Totally clueless

Ingo Oeser

PS: Who CC'ed is also subscribed to linux-mm? Or do we all filter
   dupes via "formail -D"? ;-)
-- 
10.+11.03.2001 - 3. Chemnitzer LinuxTag <http://www.tu-chemnitz.de/linux/tag>
         <<<<<<<<<<<<       come and join the fun       >>>>>>>>>>>>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Rik van Riel]

On Thu, 1 Feb 2001, Ingo Oeser wrote:
> On Thu, Feb 01, 2001 at 02:45:04PM -0200, Rik van Riel wrote:
> > One solution could be to put (most of) the swapin readahead
> > pages on the inactive_dirty list, so pressure by readahead
> > on the resident pages is smaller and the not used readahead
> > pages are reclaimed faster.
> 
> Shouldn't they be on inactive_clean anyway?

No, the inactive_clean pages are reclaimed before the
other inactive pages, and we want to give all pages
an equal chance to be used when we put them on the
inactive list.

This is especially true for freshly read in swap cache
pages, because we _expect_ that some of them will be
used.

> Or do I still not get the new linux mm design? ;-(

Read mm/swap.c::deactivate_page_nolock(), my decision to
put all clean inactive pages directly on inactive_clean
lead to the fact that dirty pages would stick around
forever and page reclaim could be quite unfair towards
clean pages. This was changed later to put all inactive
pages on the inactive_dirty list first and have them
more fairly reclaimed in page_launder.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Stephen C. Tweedie]

Hi,

On Thu, Feb 01, 2001 at 02:45:04PM -0200, Rik van Riel wrote:
> On Thu, 1 Feb 2001, Stephen C. Tweedie wrote:
> 
> But only when the extra pages we're reading in don't
> displace useful data from memory, making us fault in
> those other pages ... causing us to go to the disk
> again and do more readahead, which could potentially
> displace even more pages, etc...

Remember, it's a balance.  You can displace a few useful pages and
still win overall because the cost _per page_ goes way down due to
better disk IO utilisation.

> One solution could be to put (most of) the swapin readahead
> pages on the inactive_dirty list, so pressure by readahead
> on the resident pages is smaller and the not used readahead
> pages are reclaimed faster.

Yep, that would make much sense.

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[David Gould]

On Thu, Feb 01, 2001 at 11:26:01AM +0000, Stephen C. Tweedie wrote:
> On Wed, Jan 31, 2001 at 04:24:24PM -0800, David Gould wrote:
> > 
> > I am skeptical of the argument that we can win by replacing "the least
> > desirable" pages with pages were even less desireable and that we have
> > no recent indication of any need for. It seems possible under heavy swap
> > to discard quite a portion of the useful pages in favor of junk that just
> > happenned to have a lucky disk address.
> 
> When readin clustering was added to 2.2 for swap and paging,
> performance for a lot of VM-intensive tasks more than doubled.  Disk
> seeks are _expensive_.  If you read in 15 neighbouring pages on swapin
> and on average only one of them turns out to be useful, you have still
> halved the number of swapin IOs required.  The performance advantages
> are so enormous that easily compensate for the cost of holding the
> other, unneeded pages in memory for a while.
> 
> Also remember that the readahead pages won't actually get mapped into
> memory, so they can be recycled easily.  So, under swapping you tend
> to find that the extra readin pages are going to be replacing old,
> unneeded readahead pages to some extent, rather than swapping out
> useful pages.

Ok. I am convinced. I would have even thought of this myself eventually...

Thanks

-dg
 
-- 
David Gould                                                 dg@suse.com
SuSE, Inc.,  580 2cd St. #210,  Oakland, CA 94607          510.628.3380
You left them alone in a room with a penguin?!  Mr Gates, your men are
already dead.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] vma limited swapin readahead

[Rik van Riel]

On Thu, 1 Feb 2001, David Gould wrote:
> On Thu, Feb 01, 2001 at 11:26:01AM +0000, Stephen C. Tweedie wrote:

> > Also remember that the readahead pages won't actually get mapped into
> > memory, so they can be recycled easily.  So, under swapping you tend
> > to find that the extra readin pages are going to be replacing old,
> > unneeded readahead pages to some extent, rather than swapping out
> > useful pages.
> 
> Ok. I am convinced. I would have even thought of this myself
> eventually...

See http://distro.conectiva.com.br/bugzilla/show_bug.cgi?id=1175
for more information about this bug, and a proposed way to fix
the problem.

Or the whole Linux-MM bugzilla:
	http://www.linux-mm.org/bugzilla.shtml

cheers,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/