Re: a plea for mincore()/madvise()

Re: a plea for mincore()/madvise()

[Linus Torvalds]

On Fri, 10 Mar 2000, James Manning wrote:
> 
> I'm certainly sure there's all kinds of other areas where mincore() and
> madvise() are very helpful.

I agree especially for madvice, but I haven't liked the patches that I've
seen so far.

I don't like exporting the silly user-land "advice" into the
vma->vm_flags. I like "flags" as flags, and I'd be happy to have bit
positions saying

	if (vma->vm_flags & VM_SEQUENTIAL) {
		.. pre-fetch aggressively ..
	}


and then the madvise() system call would do somehting like

	switch (advice) {
	MADV_SEQUENTIAL:
		/* This is really more of a "mprotect" thing */
		mprotect(start, end, VM_SEQUENTIAL);
		break;
	MADV_DONTNEED:
		/* While this is really a case of msync() */
		msync(start, end, MSYNC_THROWAWAY);
		break;
	...

instead of just trying to force the madvise() system call into the VM
structure, where I don't think it makes all that much sense.

MADV_DONTNEED really is NOT a vma flag at all. It really is what Linux
tends to call MS_INVALIDATE for msync() (which is probably wrong, I don't
know how anybody else does this).

		Linus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: a plea for mincore()/madvise()

[Chuck Lever]

On Fri, 10 Mar 2000, Linus Torvalds wrote:
> I don't like exporting the silly user-land "advice" into the
> vma->vm_flags. I like "flags" as flags, and I'd be happy to have bit
> positions saying
> 
> 	if (vma->vm_flags & VM_SEQUENTIAL) {
> 		.. pre-fetch aggressively ..
> 	}

i implemented the vm flags stuff a while ago, just as you asked.  i'm not
sure my e-mail is actually getting delivered to you: i've sent the mincore
and madvise patches several times since then.

> and then the madvise() system call would do somehting like
> 
> 	switch (advice) {
> 	MADV_SEQUENTIAL:
> 		/* This is really more of a "mprotect" thing */
> 		mprotect(start, end, VM_SEQUENTIAL);
> 		break;
> 	MADV_DONTNEED:
> 		/* While this is really a case of msync() */
> 		msync(start, end, MSYNC_THROWAWAY);
> 		break;
> 	...
> 
> instead of just trying to force the madvise() system call into the VM
> structure, where I don't think it makes all that much sense.

i don't understand what you mean here.  you don't think that madvise might
have different behavior depending on what kind of vma is the target?  shm
vma's will have a different implementation than mapped file vma's.  some
types will want to implement the functionality of each MADV_ very
differently or not at all.

re-using the mprotect code for sequential, random, and normal behavior is
much preferred to what the patch does today.

	- Chuck Lever
--
corporate:	<chuckl@netscape.com>
personal:	<chucklever@netscape.net> or <cel@monkey.org>

The Linux Scalability project:
	http://www.citi.umich.edu/projects/linux-scalability/


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: a plea for mincore()/madvise()

[Linus Torvalds]

On Fri, 10 Mar 2000, Chuck Lever wrote:
> 
> i don't understand what you mean here.  you don't think that madvise might
> have different behavior depending on what kind of vma is the target?

That's not what I meant, but no, I don't think it should have different
behaviour depending on the vma anyway.

What I meant is really that the different "advices" are totally different.
MADV_DONTNEED is an operation that probably walks the page tables and just
throws the pages out (or just marks them old and uniniteresting).
Similarly MADV_WILLNEED implies more of a "start doing something now" kind
of thing. Neither would be flags in vma->vm_flags, because neither of them
are really all that much of a "save this state for future behaviour" kind
of thing.

In contrast, MADV_RANDOM is a flag that you'd set in vma->vm_flags, and
would tell the page-in logic to not pre-fetch, because it doesn't pay off.
And MADV_SEQUENTIAL would probably tell the page-in logic to pre-fetch
very aggressively. 

> re-using the mprotect code for sequential, random, and normal behavior is
> much preferred to what the patch does today.

The mprotect() routines that walk the page tables makes sense for
MADV_DONTNEED and MADV_WILLNEED. It makes no sense at all for MADV_RANDOM
and MADV_SEQUENTIAL, other than the actual vma _splitting_ part (as
opposed to the actual page table walking part).

See? I don't think the different advices are really comparable. It's
different cases.

		Linus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: a plea for mincore()/madvise()

[Chuck Lever]

On Fri, 10 Mar 2000, Linus Torvalds wrote:
> What I meant is really that the different "advices" are totally different.
> MADV_DONTNEED is an operation that probably walks the page tables and just
> throws the pages out (or just marks them old and uniniteresting).
> Similarly MADV_WILLNEED implies more of a "start doing something now" kind
> of thing. Neither would be flags in vma->vm_flags, because neither of them
> are really all that much of a "save this state for future behaviour" kind
> of thing.
> 
> In contrast, MADV_RANDOM is a flag that you'd set in vma->vm_flags, and
> would tell the page-in logic to not pre-fetch, because it doesn't pay off.
> And MADV_SEQUENTIAL would probably tell the page-in logic to pre-fetch
> very aggressively.

ja, that's exactly what my patch does.  i'm still not sure what you don't
like about it.  i'm happy to make it do anything you want, but it sounds
like we are in agreement here.

my only sticking point is that i'm not sure what community consensus there
is about MADV_DONTNEED.  i'd like to create a patch that implements
everything except MADV_DONTNEED, then maybe we should have more discussion
about exactly how that will work and add it later.

> The mprotect() routines that walk the page tables makes sense for
> MADV_DONTNEED and MADV_WILLNEED. It makes no sense at all for MADV_RANDOM
> and MADV_SEQUENTIAL, other than the actual vma _splitting_ part (as
> opposed to the actual page table walking part).

> See? I don't think the different advices are really comparable. It's
> different cases.

no argument from me, dude.  i was under the impression that mprotect only
modifies the vmflags, which is why i considered it useful for
random/sequential/normal.  haven't looked too closely at mprotect, though.

	- Chuck Lever
--
corporate:	<chuckl@netscape.com>
personal:	<chucklever@netscape.net> or <cel@monkey.org>

The Linux Scalability project:
	http://www.citi.umich.edu/projects/linux-scalability/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: a plea for mincore()/madvise()

[Linus Torvalds]

On Fri, 10 Mar 2000, Chuck Lever wrote:
> 
> ja, that's exactly what my patch does.  i'm still not sure what you don't
> like about it.  i'm happy to make it do anything you want, but it sounds
> like we are in agreement here.

Ok. I may not have seen (or noticed) your patch, I've seen earlier patches
that didn't do that.

> my only sticking point is that i'm not sure what community consensus there
> is about MADV_DONTNEED.  i'd like to create a patch that implements
> everything except MADV_DONTNEED, then maybe we should have more discussion
> about exactly how that will work and add it later.

I'd like MADV_DONTNEED to just clear the page tables. If it was a private
mapping, all the modifications get lost. If it was a shared mapping,
modifications since the last msync() may or may not get lost. 

		Linus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: a plea for mincore()/madvise()

[Chuck Lever]

On Fri, 10 Mar 2000, Linus Torvalds wrote:
> On Fri, 10 Mar 2000, Chuck Lever wrote:
> > ja, that's exactly what my patch does.  i'm still not sure what you don't
> > like about it.  i'm happy to make it do anything you want, but it sounds
> > like we are in agreement here.
> 
> Ok. I may not have seen (or noticed) your patch, I've seen earlier patches
> that didn't do that.
> 
> > my only sticking point is that i'm not sure what community consensus there
> > is about MADV_DONTNEED.  i'd like to create a patch that implements
> > everything except MADV_DONTNEED, then maybe we should have more discussion
> > about exactly how that will work and add it later.
> 
> I'd like MADV_DONTNEED to just clear the page tables. If it was a private
> mapping, all the modifications get lost. If it was a shared mapping,
> modifications since the last msync() may or may not get lost. 

i'll create a patch against 2.3.51-pre3 for madvise() and post it on the
mm list before sunday.  then we can argue about something we've both seen
:)

let me think some more about DONTNEED.  at the face of it, i agree with
your suggestion, but i may just exclude it from the patch until there is
more discussion here.

	- Chuck Lever
--
corporate:	<chuckl@netscape.com>
personal:	<chucklever@netscape.net> or <cel@monkey.org>

The Linux Scalability project:
	http://www.citi.umich.edu/projects/linux-scalability/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: a plea for mincore()/madvise()

[Christoph Rohland]

Chuck Lever <cel@monkey.org> writes:

> > I'd like MADV_DONTNEED to just clear the page tables. If it was a private
> > mapping, all the modifications get lost. If it was a shared mapping,
> > modifications since the last msync() may or may not get lost. 
> 
> i'll create a patch against 2.3.51-pre3 for madvise() and post it on the
> mm list before sunday.  then we can argue about something we've both seen
> :)
> 
> let me think some more about DONTNEED.  at the face of it, i agree with
> your suggestion, but i may just exclude it from the patch until there is
> more discussion here.

As stated before I would be very interested in a call (perhaps
MADV_DONTNEED) which throws away shared (anon) memory pages.

Greetings
		Christoph
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: a plea for mincore()/madvise()

[Richard Guenther]

On 11 Mar 2000, Christoph Rohland wrote:

> Chuck Lever <cel@monkey.org> writes:
> 
> > > I'd like MADV_DONTNEED to just clear the page tables. If it was a private
> > > mapping, all the modifications get lost. If it was a shared mapping,
> > > modifications since the last msync() may or may not get lost. 
> > 
> > i'll create a patch against 2.3.51-pre3 for madvise() and post it on the
> > mm list before sunday.  then we can argue about something we've both seen
> > :)
> > 
> > let me think some more about DONTNEED.  at the face of it, i agree with
> > your suggestion, but i may just exclude it from the patch until there is
> > more discussion here.
> 
> As stated before I would be very interested in a call (perhaps
> MADV_DONTNEED) which throws away shared (anon) memory pages.

I would like to have it throw away non anonymous shared pages, too.

Richard.

--
Richard Guenther <richard.guenther@student.uni-tuebingen.de>
WWW: http://www.anatom.uni-tuebingen.de/~richi/
The GLAME Project: http://www.glame.de/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: a plea for mincore()/madvise()

[Stephen C. Tweedie]

Hi,

On Fri, 10 Mar 2000 19:39:03 -0500 (EST), Chuck Lever <cel@monkey.org>
said:

> i'll create a patch against 2.3.51-pre3 for madvise() and post it on the
> mm list before sunday.  then we can argue about something we've both seen
> :)

> let me think some more about DONTNEED.  at the face of it, i agree with
> your suggestion, but i may just exclude it from the patch until there is
> more discussion here.

For what it's worth, Linus's request --- that DONTNEED throws stuff
away without propagating dirty data --- is something that I've had big
database folk asking us for more than once.

--Stephen


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: a plea for mincore()/madvise()

[Chuck Lever]

On Mon, 13 Mar 2000, Stephen C. Tweedie wrote:
> > let me think some more about DONTNEED.  at the face of it, i agree with
> > your suggestion, but i may just exclude it from the patch until there is
> > more discussion here.
> 
> For what it's worth, Linus's request --- that DONTNEED throws stuff
> away without propagating dirty data --- is something that I've had big
> database folk asking us for more than once.

i believe that allowing an application to dispose of data this way is a
good idea, and that it's something that Linux should allow.  but there was
an interesting discussion here on linux-mm recently that described a
slightly different interface that would allow this and some other
functionality too.  i'd just like to study the suggestions a bit.

the rest of madvise is already implemented, i just have to port it to
2.3.51.  i'll post that piece of it soon.

	- Chuck Lever
--
corporate:	<chuckl@netscape.com>
personal:	<chucklever@netscape.net> or <cel@monkey.org>

The Linux Scalability project:
	http://www.citi.umich.edu/projects/linux-scalability/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/