Syncing the page cache, take 2

Syncing the page cache, take 2

[Daniel Phillips]

My earlier attempt to state the problem was not as clear as it could have been.

This is really a VFS problem not a mm problem per se, but since the two have
become so closely intertwined I'm bringing it up here.

There seems to be something missing in the current VFS (please correct me if I'm
wrong): the sync code totally ignores the page cache, so when you do a sync
you're only syncing the buffer cache and not file data that may have been mapped
into the page cache by file_write or file_mmap.

OK, if that is indeed the case then it needs to be fixed.

Since it needs to be fixed then this is a good time to state a particular need
that I have in my filesystem project: for optimal operation I need to be able to
sync the page cache to the buffer cache selectively.  An appropriate granularity
would be per-mapping.  So I would like to have a VFS call something like:

    void write_mapping_now (struct address_space *mapping, int sync)

This is modelled on the write_inode_now function.  (Not that I think that's the
greatest name in the world - I'm just trying to continue the pattern.)  The
exact semantics of write_mapping_now would likely be subject to considerable
discussion, but its effect on the page case is clear: for every dirty page in
the 
mapping address_space_operations->writepage should be called once.  This would
give me the syncing ability I need.

Arguably, write_mapping_now should be called by write_inode_now.

--
Daniel



-- 
Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Syncing the page cache, take 2

[Stephen C. Tweedie]

Hi,

On Tue, Aug 15, 2000 at 10:32:14AM +0200, Daniel Phillips wrote:
> 
> This is really a VFS problem not a mm problem per se, but since the two have
> become so closely intertwined I'm bringing it up here.
> 
> There seems to be something missing in the current VFS (please correct me if I'm
> wrong): the sync code totally ignores the page cache, so when you do a sync
> you're only syncing the buffer cache and not file data that may have been mapped
> into the page cache by file_write or file_mmap.

Correct.  We have plans to change this in 2.5, basically by removing
the VM's privileged knowledge about the buffer cache and making the
buffer operations (write-back, unmap etc.) into special cases of
generic address-space operations.  For 2.4, it's really to late to do
anything about this.

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Syncing the page cache, take 2

[Rik van Riel]

On Tue, 15 Aug 2000, Stephen C. Tweedie wrote:

> Correct.  We have plans to change this in 2.5, basically by
> removing the VM's privileged knowledge about the buffer cache
> and making the buffer operations (write-back, unmap etc.) into
> special cases of generic address-space operations.  For 2.4,
> it's really to late to do anything about this.

Stephen,

please take a look at my VM patch at http://www.surriel.com/patches/
(either the -test4 or the -test7-pre4 one).

If you look closely at mm/vmscan.c::page_launder(), you'll see
that we should be able to add the flush callback with only about
5 to 10 lines of changed code ...

(and even more ... we just about *need* the flush callback when
we're running in a multi-queue VM)

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Syncing the page cache, take 2

[Stephen C. Tweedie]

Hi,

On Wed, Aug 16, 2000 at 12:14:33AM +0200, Daniel Phillips wrote:
> > 
> > (and even more ... we just about *need* the flush callback when
> > we're running in a multi-queue VM)
> 
> OK, but what about the case where the filesystem knows it wants the page
> cache to flush *right now*?  For example, when a filesystem wants to be
> sure the page cache is synced through to buffers just before marking a
> consistent state in the journal, say.  How does it make that happen?

Correct --- remember, Rik, that we talked about this?  It's not just
enough for the VM to call the address-space to flush dirty pages: you
also need to delegate the ability to manipulate the dirty status to
the fs.  In other words, you need a mark_page_dirty/clean() for pages
just as you already have for buffers.  

Do that and the filesystems can do pretty much what they want in
response to the callback.

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Syncing the page cache, take 2

[Rik van Riel]

On Wed, 16 Aug 2000, Stephen C. Tweedie wrote:
> On Wed, Aug 16, 2000 at 12:14:33AM +0200, Daniel Phillips wrote:
> > > 
> > > (and even more ... we just about *need* the flush callback when
> > > we're running in a multi-queue VM)
> > 
> > OK, but what about the case where the filesystem knows it wants the page
> > cache to flush *right now*?  For example, when a filesystem wants to be
> > sure the page cache is synced through to buffers just before marking a
> > consistent state in the journal, say.  How does it make that happen?
> 
> Correct --- remember, Rik, that we talked about this?  It's not just
> enough for the VM to call the address-space to flush dirty pages: you
> also need to delegate the ability to manipulate the dirty status to
> the fs.  In other words, you need a mark_page_dirty/clean() for pages
> just as you already have for buffers.  
> 
> Do that and the filesystems can do pretty much what they want in
> response to the callback.

Indeed. I'll get to work on this right after the new VM patch
is tuned to a level where it works fine for everybody.

(I need to add something to keep the pagecache small during
very heavy, use-once, IO)

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Syncing the page cache, take 2

[Rik van Riel]

On Wed, 16 Aug 2000, Daniel Phillips wrote:
> Rik van Riel wrote:
> > On Tue, 15 Aug 2000, Stephen C. Tweedie wrote:
> > 
> > > Correct.  We have plans to change this in 2.5, basically by
> > > removing the VM's privileged knowledge about the buffer cache
> > > and making the buffer operations (write-back, unmap etc.) into
> > > special cases of generic address-space operations.  For 2.4,
> > > it's really to late to do anything about this.
> > 
> > please take a look at my VM patch at http://www.surriel.com/patches/
> > (either the -test4 or the -test7-pre4 one).
> > 
> > If you look closely at mm/vmscan.c::page_launder(), you'll see
> > that we should be able to add the flush callback with only about
> > 5 to 10 lines of changed code ...
> > 
> > (and even more ... we just about *need* the flush callback when
> > we're running in a multi-queue VM)
> 
> OK, but what about the case where the filesystem knows it wants
> the page cache to flush *right now*?  For example, when a
> filesystem wants to be sure the page cache is synced through to
> buffers just before marking a consistent state in the journal,
> say.  How does it make that happen?

There are some ugly tricks, like pinning the buffer so that
the buffer flushing code can't flush it out. Most of these
have the potential for messing up VM and making the box run
out of memory or even making the box hang...

If the new VM makes it into 2.4, however, it should be
relatively easy to add the flush callback as a function
in page_launder(). Otherwise we could add yet another
thing to shrink_mmap() ... we could do it, but I guess
at the cost of some code readability  *grin*

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/