Re: 2.6.19-rc1-mm1

Re: 2.6.19-rc1-mm1

[Andrew Morton]

On Tue, 10 Oct 2006 09:20:00 +0200
Arjan van de Ven <arjan@infradead.org> wrote:

> On Tue, 2006-10-10 at 00:09 -0700, Andrew Morton wrote:
> > +htlb-forget-rss-with-pt-sharing.patch

Which I didn't write.  cc's added.

> if it's ok to ignore RSS,

We'd prefer not to.  But what's the alternative?

> can we consider the shared pagetables for
> normal pages patch?

Has been repeatedly considered, but Hugh keeps finding bugs in it.

> It saves quite a bit of memory on even desktop
> workloads as well as avoiding several (soft) pagefaults.
> 
> So.. what does RSS actually mean? Can we ignore it somewhat for
> shared-readonly mappings ? 

We'd prefer to go the other way, and implement RLIMIT_RSS wouldn't we?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.19-rc1-mm1

[Arjan van de Ven]

On Tue, 2006-10-10 at 00:45 -0700, Andrew Morton wrote:

> > if it's ok to ignore RSS,
> 
> We'd prefer not to.  But what's the alternative?

it's a good question; today (2.6.18) we have some defacto behavior of
RSS; 2.6.19-rc1-mm1 has a somewhat different one. Either can be entirely
valid; and we can obviously implement either. We can go even further and
remove more from RSS to help save memory and pagefaults (both help
desktop performance) by going the shared pagetable road
> 
> > can we consider the shared pagetables for
> > normal pages patch?
> 
> Has been repeatedly considered, but Hugh keeps finding bugs in it.

the latest one I tried looked relatively simple (earlier ones were very
complex) so maybe Hugh can find time to give it another lookover?

> 
> > It saves quite a bit of memory on even desktop
> > workloads as well as avoiding several (soft) pagefaults.
> > 
> > So.. what does RSS actually mean? Can we ignore it somewhat for
> > shared-readonly mappings ? 
> 
> We'd prefer to go the other way, and implement RLIMIT_RSS wouldn't we?

Well... that again depends on how we define RSS. implementing the rlimit
doesn't mean we can't NOT count certain things (like the hugetlb pages
in the patch above, or shared read only pagecache pages) to be part of
it. It's a fundamental "what does it mean" thing.
You can argue that RSS means "all memory that the application has in
it's address space", you can argue "all such memory except a few cases",
you can argue "all memory that is private/exclusive to the
application"... 
This is not a pointless piss-in-the-wind discussion; unless we define
rather specific what it really means, the RLIMIT doesn't mean anything
either.

We need to consider at least if any of the following are part of rss:
* VM_IO io mmaped device stuff 
* Non-linear mappings
* Shared hugetlb memory that shares pagetables
* Shared hugetlb memory
* Hugetlb memory in general
* Shared normal memory that shares pagetables
* Shared normal memory (file backed; eg pagecache)
* Shared normal memory (anonymous/non-file-backed)
* Sysv/ipc shared memory
* Not shared normal memory

I don't think posix or anything else helps us here so we can vote or
otherwise reason which make sense and which don't. I hope the outcome is
reasonably consistent ;)

I know the desktop guys at least consider RSS useless as measure of "how
much memory does my desktop app take"; especially since they have many
shared libraries and they consider it unfair that each app pays the full
price in terms of RSS for those. So personally I'm not unhappy with a
definition that comes down to "all memory that's private to the app";
although it is a change from what 2.6.18 does.



-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RSS accounting (was: Re: 2.6.19-rc1-mm1)

[Peter Zijlstra]

On Tue, 2006-10-10 at 10:03 +0200, Arjan van de Ven wrote:
> On Tue, 2006-10-10 at 00:45 -0700, Andrew Morton wrote:
> 
> > > if it's ok to ignore RSS,
> > 
> > We'd prefer not to.  But what's the alternative?
> 
> it's a good question; today (2.6.18) we have some defacto behavior of
> RSS; 2.6.19-rc1-mm1 has a somewhat different one. Either can be entirely
> valid; and we can obviously implement either. We can go even further and
> remove more from RSS to help save memory and pagefaults (both help
> desktop performance) by going the shared pagetable road

> > > So.. what does RSS actually mean? Can we ignore it somewhat for
> > > shared-readonly mappings ? 
> > 
> > We'd prefer to go the other way, and implement RLIMIT_RSS wouldn't we?
> 
> Well... that again depends on how we define RSS. implementing the rlimit
> doesn't mean we can't NOT count certain things (like the hugetlb pages
> in the patch above, or shared read only pagecache pages) to be part of
> it. It's a fundamental "what does it mean" thing.
> You can argue that RSS means "all memory that the application has in
> it's address space", you can argue "all such memory except a few cases",
> you can argue "all memory that is private/exclusive to the
> application"... 
> This is not a pointless piss-in-the-wind discussion; unless we define
> rather specific what it really means, the RLIMIT doesn't mean anything
> either.
> 
> We need to consider at least if any of the following are part of rss:
> * VM_IO io mmaped device stuff 
> * Non-linear mappings
> * Shared hugetlb memory that shares pagetables
> * Shared hugetlb memory
> * Hugetlb memory in general
> * Shared normal memory that shares pagetables
> * Shared normal memory (file backed; eg pagecache)
> * Shared normal memory (anonymous/non-file-backed)
> * Sysv/ipc shared memory
> * Not shared normal memory
> 
> I don't think posix or anything else helps us here so we can vote or
> otherwise reason which make sense and which don't. I hope the outcome is
> reasonably consistent ;)
> 
> I know the desktop guys at least consider RSS useless as measure of "how
> much memory does my desktop app take"; especially since they have many
> shared libraries and they consider it unfair that each app pays the full
> price in terms of RSS for those. So personally I'm not unhappy with a
> definition that comes down to "all memory that's private to the app";
> although it is a change from what 2.6.18 does.

So we have three cases where RSS matters besides presenting a number to
user-space;
 - shared page tables
 - containers
 - rlimit

Preferably they will all share a common definition of what RSS is; since
containers must account shared pages somehow (not doing so would open up
a large hole to DoS the other containers) and the container case can be
argued to be an extension of the rlimit case we cannot just ignore them.

As then what to do with them, I've yet to figure out. Some random bit
floating in my brain;
 - VM_IO can be discarted, its not actual memory

 - hugetlb is memory too, so I'd not special case this (other than the
different unit of accounting)

 - shared mapped pages could be accounted on vma level, since both
containers have access to the same file, there is already an imbalance,
so I'd not worry about the 1%-99% usage scenario here.

 - regular, non mapped, pagecache pages however have no owner - what to
do. (fake vma - which would result in each container paying equally for
all these pages?)

Anyway, I'd rather not break RSS twice, once now because we don't quite
know what to do, and later when we do get an acceptable mm container and
have to include shared memory in one way or the other.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RSS accounting (was: Re: 2.6.19-rc1-mm1)

[Arjan van de Ven]

On Tue, 2006-10-10 at 15:14 +0200, Peter Zijlstra wrote:
> > 
> > We need to consider at least if any of the following are part of rss:
> > * VM_IO io mmaped device stuff 
> > * Non-linear mappings
> > * Shared hugetlb memory that shares pagetables
> > * Shared hugetlb memory
> > * Hugetlb memory in general
> > * Shared normal memory that shares pagetables
> > * Shared normal memory (file backed; eg pagecache)
> > * Shared normal memory (anonymous/non-file-backed)
> > * Sysv/ipc shared memory
> > * Not shared normal memory


> So we have three cases where RSS matters besides presenting a number to
> user-space;
>  - shared page tables
>  - containers
>  - rlimit
> 
> Preferably they will all share a common definition of what RSS is; since
> containers must account shared pages somehow (not doing so would open up
> a large hole to DoS the other containers) and the container case can be
> argued to be an extension of the rlimit case we cannot just ignore them.
> 
> As then what to do with them, I've yet to figure out. Some random bit
> floating in my brain;
>  - VM_IO can be discarted, its not actual memory

agreed (although I think we do count it today; this is half of what
makes X look so bloated, next to firefox ;)

> 
>  - hugetlb is memory too, so I'd not special case this (other than the
> different unit of accounting)

agreed again; personally I don't think hugetlb memory should be special;
especially with all the libhugetlbfs work on making it real easy for
apps to use; the more it's used the more people would notice something
funky with it.


>  - shared mapped pages could be accounted on vma level, since both
> containers have access to the same file, there is already an imbalance,
> so I'd not worry about the 1%-99% usage scenario here.

or one level below; if you count it in the actual PTE page then the
sharing case will "just work". It's a trick question; do you count it as
100% or do you count it as 100% / number of sharers.


>  - regular, non mapped, pagecache pages however have no owner - what to
> do. (fake vma - which would result in each container paying equally for
> all these pages?)

if they're clean I wouldn't count them to anything actually


> Anyway, I'd rather not break RSS twice, once now because we don't quite
> know what to do, and later when we do get an acceptable mm container and
> have to include shared memory in one way or the other.


RSS of a container versus RSS of a process is an interesting question
for sure ;)


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RSS accounting (was: Re: 2.6.19-rc1-mm1)

[Eric W. Biederman]

Arjan van de Ven <arjan@infradead.org> writes:

> On Tue, 2006-10-10 at 15:14 +0200, Peter Zijlstra wrote:
>> > 
>> > We need to consider at least if any of the following are part of rss:
>> > * VM_IO io mmaped device stuff 
>> > * Non-linear mappings
>> > * Shared hugetlb memory that shares pagetables
>> > * Shared hugetlb memory
>> > * Hugetlb memory in general
>> > * Shared normal memory that shares pagetables
>> > * Shared normal memory (file backed; eg pagecache)
>> > * Shared normal memory (anonymous/non-file-backed)
>> > * Sysv/ipc shared memory
>> > * Not shared normal memory

There is a concept related to RSS that is very interesting.  The
minimum RSS that a process needs to keep from thrashing.

It can be shown that if your RSS rlimit is greater that your minimum
RSS your process will never thrash. 

One thing that older paging algorithms would try and do when there
was memory pressure was to dynamically discover an applications
minimum RSS, and if they couldn't meet it swap that process out,
because the program can't make progress anyway.

A per process not a per application RSS fails to model multiple
process applications but the concepts are sound.

So since VM_IO does not have any effect on paging it should not
be counted but if you were a stickler the page tables from the
VM_IO should be counted.

The other thing to note is even if you RSS is just above your minimum
RSS that will only result in your process having pages bounce in and
out of the page cache (not necessarily to disk).   So while it will
increase minor faults a restrictive RSS is not a problem when
you have excess resources.

>>  - shared mapped pages could be accounted on vma level, since both
>> containers have access to the same file, there is already an imbalance,
>> so I'd not worry about the 1%-99% usage scenario here.
>
> or one level below; if you count it in the actual PTE page then the
> sharing case will "just work". It's a trick question; do you count it as
> 100% or do you count it as 100% / number of sharers.

I'm not certain I even want to follow the logic here. The answer is
clear.

For processes shared pages are not special.

For computing a container RSS shared pages need to be counted the
first time they are mapped by any process in a container, and
uncounted the last time they are unmapped by a process in a container.
With rmap we have the data structures necessary to do the accounting,
although it might be a bit of a pain.


>>  - regular, non mapped, pagecache pages however have no owner - what to
>> do. (fake vma - which would result in each container paying equally for
>> all these pages?)
>
> if they're clean I wouldn't count them to anything actually

If the pages aren't mapped they aren't part of the resident set size.

At least not until we start worrying about per container or per
process splits of the page cache, and there are clearly good reasons
to avoid that.

>> Anyway, I'd rather not break RSS twice, once now because we don't quite
>> know what to do, and later when we do get an acceptable mm container and
>> have to include shared memory in one way or the other.
>
>
> RSS of a container versus RSS of a process is an interesting question
> for sure ;)

Agreed.  Historically it was much more interesting because we didn't
have rmap so telling if your set of processes had mapped the page
already is a challenge.  At this point unless the performance of
the accounting is to much it should just be a simple counting problem.

The real challenge is doing a decent job of picking the appropriate
page to unmap when a new mapping by a process would exceed the rss
limit.  That is the one piece of the puzzle we have never implemented
:(

You can declare success when you can push one container into heavy
swapping and the rest of the containers are still running fine.

Eric

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RSS accounting (was: Re: 2.6.19-rc1-mm1)

[Arjan van de Ven]

On Tue, 2006-10-10 at 17:54 -0600, Eric W. Biederman wrote:

> For processes shared pages are not special.

depends on what question you want to answer with RSS.
If the question is "workload working set size" then you are right. If
the question is "how much ram does my application cause to be used" the
answer is FAR less clear....

You seem to have an implicit definition on what RSS should mean; but
it's implicit. Mind making an explicit definition of what RSS should be
in your opinion? I think that's the biggest problem we have right now;
several people have different ideas about what it should/could be, and
as such we're not talking about the same thing. Lets first agree/specify
what it SHOULD mean, and then we can figure out what gets counted for
that ;)


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RSS accounting (was: Re: 2.6.19-rc1-mm1)

[Eric W. Biederman]

Arjan van de Ven <arjan@infradead.org> writes:

> On Tue, 2006-10-10 at 17:54 -0600, Eric W. Biederman wrote:
>
>> For processes shared pages are not special.

Actually the above is not quite true you can map a shared page twice
into the same process but in practice it rarely happens.

> depends on what question you want to answer with RSS.
> If the question is "workload working set size" then you are right. If
> the question is "how much ram does my application cause to be used" the
> answer is FAR less clear....

There are two basic concerns.  How do you keep an application from
going crazy and trashing the rest of your system?  A question on what
number do you need to implement a resource limit.

The other question is how do I get good information so I can
effectively understand what kind of resources a given
application is using and hopefully predict what kind of
resources that application will use in the future.

The last time I tried to answer the question "how much ram does my
application cause to be used"  I had to slowly start up additional
copies and watch how much free memory decreased.

Having a couple of additional counts in addition to RSS would probably
be the most help in understanding resource usage.  Counting the number
of private dirty resident pages would be interesting.  As would
counting the number of pages that are resident in the process but not
resident in any other process.

It might also help to have a per page report on which file it backs
and how many user it has.  Unfortunately that is totally overwhelming
detail and the act of reporting it would quite likely change the
result as it would take so much memory to store the result.

> You seem to have an implicit definition on what RSS should mean; but
> it's implicit. Mind making an explicit definition of what RSS should be
> in your opinion? I think that's the biggest problem we have right now;
> several people have different ideas about what it should/could be, and
> as such we're not talking about the same thing. Lets first agree/specify
> what it SHOULD mean, and then we can figure out what gets counted for
> that ;)

Well I tried to defined it in terms of what you can use it for.

I would define the resident set size as the total number of bytes
of physical RAM that a process (or set of processes) is using,
irrespective of the rest of the system.  

By physical RAM I mean that if a single page (if shared) is used twice
by a single process it will be counted only once.  This definition
works well for shared and private pages.  

COW pages are probably the most subtle.  They are both shared and not
shared.   Since that sharing is application visible and at application
discretion I would count COW pages as shared until they that sharing
is broken, and then I would count them as private pages.  

The principle is that you don't find the ``owner'' of a page and
charge the page to the ``owner''.  Instead you find the users
of a page and charge all of them for the page exactly once.

By and large most of that usage comes from pages in the page tables
so they are the most interesting items to count.

Things like file descriptors, inodes, page tables and other kernel
memory are interesting but are generally overshadowed by the page
table users, and generally kernel data structures don't have reverse
maps which makes it difficult to charge all of the users.

So I think the counting should be primarily about what is mapped into
the page tables.  But other things can be added as is appropriate or
easy.

The practical effect should be that an application that needs more
pages than it's specified RSS to avoid thrashing should thrash but
it shouldn't take the rest of the system with it.


The biggest instance of system memory that an application does not
seem to have true control over is the page cache.  Some kind of limit
that prevents one application from destroying everything another
application doing seems interesting.  But I expect the solution there
are I/O limits and not memory limits.

Eric

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RSS accounting (was: Re: 2.6.19-rc1-mm1)

[Arjan van de Ven]

On Wed, 2006-10-11 at 06:07 -0600, Eric W. Biederman wrote:
> Arjan van de Ven <arjan@infradead.org> writes:
> 
> > On Tue, 2006-10-10 at 17:54 -0600, Eric W. Biederman wrote:
> >
> >> For processes shared pages are not special.
> 
> Actually the above is not quite true you can map a shared page twice
> into the same process but in practice it rarely happens.

yeah I'm entirely fine with ignoring that case (or making the person who
does it pay for it :)

> 
> > depends on what question you want to answer with RSS.
> > If the question is "workload working set size" then you are right. If
> > the question is "how much ram does my application cause to be used" the
> > answer is FAR less clear....
> 
> There are two basic concerns.  How do you keep an application from
> going crazy and trashing the rest of your system?  A question on what
> number do you need to implement a resource limit.

yet at the same time if 2 apps mmap a shared file, and app 1 keeps it in
pagecache, it doesn't cause app2 to trash, or rather, it's not like if
app 2 did NOT have the page from that file, the system wouldn't trash.


> > You seem to have an implicit definition on what RSS should mean; but
> > it's implicit. Mind making an explicit definition of what RSS should be
> > in your opinion? I think that's the biggest problem we have right now;
> > several people have different ideas about what it should/could be, and
> > as such we're not talking about the same thing. Lets first agree/specify
> > what it SHOULD mean, and then we can figure out what gets counted for
> > that ;)
> 
> Well I tried to defined it in terms of what you can use it for.
> 
> I would define the resident set size as the total number of bytes
> of physical RAM that a process (or set of processes) is using,
> irrespective of the rest of the system.  
>  
> 
> So I think the counting should be primarily about what is mapped into
> the page tables.  But other things can be added as is appropriate or
> easy.
> 
> The practical effect should be that an application that needs more
> pages than it's specified RSS to avoid thrashing should thrash but
> it shouldn't take the rest of the system with it.


so by your definition, hugepages are part of RSS.

Ken: what is your definition of RSS ?


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: RSS accounting (was: Re: 2.6.19-rc1-mm1)

[Chen, Kenneth W]

Arjan van de Ven wrote on Wednesday, October 11, 2006 6:55 AM
> > Well I tried to defined it in terms of what you can use it for.
> > 
> > I would define the resident set size as the total number of bytes
> > of physical RAM that a process (or set of processes) is using,
> > irrespective of the rest of the system.  
> > 
> > So I think the counting should be primarily about what is mapped into
> > the page tables.  But other things can be added as is appropriate or
> > easy.
> > 
> > The practical effect should be that an application that needs more
> > pages than it's specified RSS to avoid thrashing should thrash but
> > it shouldn't take the rest of the system with it.
> 
> 
> so by your definition, hugepages are part of RSS.
> 
> Ken: what is your definition of RSS ?

I'm more inclined to define RSS as "how much ram does my application
cause to be used".  To monitor process's working set size, We already
have /proc/<pid>/smaps.  Whether we can use working set size in an
intelligent way in mm is an interesting question. Though, so far such
accounting is not utilized at all.

- Ken

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RSS accounting (was: Re: 2.6.19-rc1-mm1)

[Benjamin LaHaise]

On Wed, Oct 11, 2006 at 10:15:39AM -0700, Chen, Kenneth W wrote:
> I'm more inclined to define RSS as "how much ram does my application
> cause to be used".  To monitor process's working set size, We already
> have /proc/<pid>/smaps.  Whether we can use working set size in an
> intelligent way in mm is an interesting question. Though, so far such
> accounting is not utilized at all.

If that is the case, it would make sense to account such things as page 
tables and other kernel allocations against the RSS, which would be useful.  
That said, it's possible to keep semantics fairly close to those currently 
implemented by tracking RSS differently for shared vs private areas -- 
those vmas which are shared could be placed on a list and then summed when 
RSS is read.  That said, I'm not sure it is a good idea, as the cost of 
obtaining RSS for tools like top is exactly why we have the current 
counters maintained to provide O(1) semantics.

All of the old semantics are covered by smaps, though, so I'd agree with 
any changes to make RSS reflect allocations incurred by this process.

		-ben
-- 
"Time is of no importance, Mr. President, only life is important."
Don't Email: <dont@kvack.org>.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>