Re: how text page of executable are shared ?

Re: how text page of executable are shared ?

[pnilesh]

Hi,

No, the page count will be three at least.  The presence of the page
in the page cache counts as one, and both of the page-table mappings of
the page each count as a further reference.

So when no process is pointing to a page in page cache the count will be
one.
But what is the difference if we have this to zero any way it is not being
refernced by any process.
Or can we have a page cache entry with page count as zero ?

Also all the pages which are present in the memory for any process will
also be part of the page hash queue and if they belong to a file then they
will also be on the inode queue.
Am I right.




Yes.  swap_out() is responsible for unlinking pages from process page
tables.  In the case you describe, the page will still have outstanding
references, from the other process and from the page cache.  Only when
the page cache cleanup function (shrink_mmap) gets called, after all of
the ptes to the page have been cleared, will the page be freed.

If you think about it, this is natural: when a process pages in a binary
and then exits, we really want the pages still to remain in memory so
that if you immediately rerun the program, we don't have to go back to
disk for the pages.  The process exiting acts a bit like a complete
swap_out, freeing up the pte reference to the page, but the page still
remains in the page cache until the memory is needed for something else.

> Q    When a page of a file is in page hash queue, does this page have
page
> table entry in any process ?

It may have, but it doesn't have to.

> Q     Can this be discarded right away , if the need arises?

Not without first doing a swap_out() on all the references to the page.
The Linux VM does its swapout based on virtual, not physical, page
scanning (although shrink_mmap() is physical).

--Stephen



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: how text page of executable are shared ?

[Stephen C. Tweedie]

Hi,

On Wed, Mar 29, 2000 at 01:16:37PM +0530, pnilesh@in.ibm.com wrote:
> 
> So when no process is pointing to a page in page cache the count will be
> one.
> But what is the difference if we have this to zero any way it is not being
> refernced by any process.
> Or can we have a page cache entry with page count as zero ?

Pages are returned to the system free list as soon as the count reaches
zero.  The swapper does not do that, though: swapped pages are always
entered into the page cache through the swap cache mechanism, and are
finally freed from there.

> Also all the pages which are present in the memory for any process will
> also be part of the page hash queue and if they belong to a file then they
> will also be on the inode queue.

No, anonymous data pages are not usually in the page cache at all.  They
only ever lie in the page cache if there is swapping going on (the VM
effectively treats swapping as a forced mmap() of a page of swap).

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: how text page of executable are shared ?

[Stephen C. Tweedie]

Hi,

On Wed, Mar 29, 2000 at 05:21:21AM +0000, Andrew Morton wrote:
> > 
> > The swapping should be brief if all is working properly, though, as the
> > shrink_mmap() will rapidly find itself on the second pass over memory
> > and will start finding things which have been aged on the first pass
> > and not used since.
> 
> Interesting.
> 
> Why do you swap active pages out (page_count(page) > 1) when there are
> still (page_count(page) == 1) pages floating about?

We don't.  Scanning the page cache for unreferenced, count==1 pages 
always takes priority over swapping.

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: how text page of executable are shared ?

[Andrew Morton]

"Stephen C. Tweedie" wrote:
> 
> Hi,
> 
> On Tue, Mar 28, 2000 at 10:58:00AM -0500, Mark Hahn wrote:
> >
> > could you comment on a problem I'm seeing in the current (pre3) VM?
> > the situation is a 256M machine, otherwise idle (random daemons, no X,
> > couple ssh's) and a process that sequentially traverses 12 40M files
> > by mmaping them (and munmapping them, in order, one at a time.)
> >
> > the observation is that all goes well until the ~6th file, when we
> > run out of unused ram.  then we start _swapping_!  the point is that
> > shrink_mmap should really be scavenging those now unmapped files,
> > shouldn't it?
> 
> Well, you've filled the whole of memory with recently referenced page
> cache pages.  The page cache scanner can now scan the whole of physical
> memory without finding anything which is "old" enough to be evicted.
> It is only natural that it will start swapping at that point!
> 
> The swapping should be brief if all is working properly, though, as the
> shrink_mmap() will rapidly find itself on the second pass over memory
> and will start finding things which have been aged on the first pass
> and not used since.

Interesting.

Why do you swap active pages out (page_count(page) > 1) when there are
still (page_count(page) == 1) pages floating about?


-- 
-akpm-
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: how text page of executable are shared ?

[Kanoj Sarcar]

> 
> Hi,
> 
> On Tue, Mar 28, 2000 at 10:58:00AM -0500, Mark Hahn wrote:
> > 
> > could you comment on a problem I'm seeing in the current (pre3) VM?
> > the situation is a 256M machine, otherwise idle (random daemons, no X,
> > couple ssh's) and a process that sequentially traverses 12 40M files
> > by mmaping them (and munmapping them, in order, one at a time.)
> > 
> > the observation is that all goes well until the ~6th file, when we 
> > run out of unused ram.  then we start _swapping_!  the point is that 
> > shrink_mmap should really be scavenging those now unmapped files,
> > shouldn't it?
> 
> Well, you've filled the whole of memory with recently referenced page 
> cache pages.  The page cache scanner can now scan the whole of physical
> memory without finding anything which is "old" enough to be evicted. 
> It is only natural that it will start swapping at that point!
> 
> The swapping should be brief if all is working properly, though, as the
> shrink_mmap() will rapidly find itself on the second pass over memory
> and will start finding things which have been aged on the first pass 
> and not used since.
> 
> --Stephen
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux.eu.org/Linux-MM/
> 

As I mentioned to Mark in private mail, it might be worthwhile looking
into the possiblity of using MADV_DONTNEED to discard file pages from 
the cache. In some os'es, I think msync(MS_INVALIDATE) actually takes
the page out from the cache.

Kanoj
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: how text page of executable are shared ?

[Stephen C. Tweedie]

Hi,

On Tue, Mar 28, 2000 at 10:58:00AM -0500, Mark Hahn wrote:
> 
> could you comment on a problem I'm seeing in the current (pre3) VM?
> the situation is a 256M machine, otherwise idle (random daemons, no X,
> couple ssh's) and a process that sequentially traverses 12 40M files
> by mmaping them (and munmapping them, in order, one at a time.)
> 
> the observation is that all goes well until the ~6th file, when we 
> run out of unused ram.  then we start _swapping_!  the point is that 
> shrink_mmap should really be scavenging those now unmapped files,
> shouldn't it?

Well, you've filled the whole of memory with recently referenced page 
cache pages.  The page cache scanner can now scan the whole of physical
memory without finding anything which is "old" enough to be evicted. 
It is only natural that it will start swapping at that point!

The swapping should be brief if all is working properly, though, as the
shrink_mmap() will rapidly find itself on the second pass over memory
and will start finding things which have been aged on the first pass 
and not used since.

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: how text page of executable are shared ?

[Andrea Arcangeli]

On Tue, 28 Mar 2000, Mark Hahn wrote:

>in shrink_mmap:
>                /*
>                 * We can't free pages unless there's just one user
>                 * (count == 2 because we added one ourselves above).
>                 */
>                if (page_count(page) != 2)
>                        goto cache_unlock_continue;
>
>is this wrong, since the page cache holds a reference?

That is right. At that point the miniumum count is 2 (there you could
say also:

	if (page_count < 2)
		BUG()

if you want).

1 reference is in the page cache and 1 reference is for the additional
get_page that shrink_mmap does to avoid the page to be freed under it.
Thus if the page count is 2 we can go ahead and release the page. If the
page count is != 2 (that means >2) we can't play with such page since
somebody else is using it.

Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: how text page of executable are shared ?

[Mark Hahn]

> > The page table entries of both the process will have entry for this page.
> > But when the page is discarded only the page entry of only one process get
> > cleared , this is what I have understood from the swap_out () function .
> 
> Yes.  swap_out() is responsible for unlinking pages from process page 
> tables.  In the case you describe, the page will still have outstanding
> references, from the other process and from the page cache.  Only when
> the page cache cleanup function (shrink_mmap) gets called, after all of
> the ptes to the page have been cleared, will the page be freed.

could you comment on a problem I'm seeing in the current (pre3) VM?
the situation is a 256M machine, otherwise idle (random daemons, no X,
couple ssh's) and a process that sequentially traverses 12 40M files
by mmaping them (and munmapping them, in order, one at a time.)

the observation is that all goes well until the ~6th file, when we 
run out of unused ram.  then we start _swapping_!  the point is that 
shrink_mmap should really be scavenging those now unmapped files,
shouldn't it?

could something be happening, like we're accidentally setting PG_referenced
on pages that are only in use by the page cache?  or perhaps someone not
adjusting page_cache_size properly?

in shrink_mmap:
                /*
                 * We can't free pages unless there's just one user
                 * (count == 2 because we added one ourselves above).
                 */
                if (page_count(page) != 2)
                        goto cache_unlock_continue;

is this wrong, since the page cache holds a reference?


thanks, mark hahn.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: how text page of executable are shared ?

[Stephen C. Tweedie]

Hi,

On Tue, Mar 28, 2000 at 09:21:59AM +0530, pnilesh@in.ibm.com wrote:
> 
> Suppose a text page of an executable is mapped in the address space of 2
> processes. The page count will be one.

No, the page count will be three at least.  The presence of the page 
in the page cache counts as one, and both of the page-table mappings of
the page each count as a further reference.

> The page table entries of both the process will have entry for this page.
> But when the page is discarded only the page entry of only one process get
> cleared , this is what I have understood from the swap_out () function .

Yes.  swap_out() is responsible for unlinking pages from process page 
tables.  In the case you describe, the page will still have outstanding
references, from the other process and from the page cache.  Only when
the page cache cleanup function (shrink_mmap) gets called, after all of
the ptes to the page have been cleared, will the page be freed.

If you think about it, this is natural: when a process pages in a binary
and then exits, we really want the pages still to remain in memory so 
that if you immediately rerun the program, we don't have to go back to
disk for the pages.  The process exiting acts a bit like a complete
swap_out, freeing up the pte reference to the page, but the page still
remains in the page cache until the memory is needed for something else.

> Q    When a page of a file is in page hash queue, does this page have page
> table entry in any process ?

It may have, but it doesn't have to.

> Q     Can this be discarded right away , if the need arises?

Not without first doing a swap_out() on all the references to the page.
The Linux VM does its swapout based on virtual, not physical, page
scanning (although shrink_mmap() is physical).

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

how text page of executable are shared ?

[pnilesh]

     When a executable file runs there is only one copy of the text part in
the memory. But I have some doubts as I am not able to figure how exactly
this is done.

Suppose a text page of an executable is mapped in the address space of 2
processes. The page count will be one.
The page table entries of both the process will have entry for this page.
But when the page is discarded only the page entry of only one process get
cleared , this is what I have understood from the swap_out () function .
But the page table entry of the other process is still pointing to the page
which has been discarded.


Can any body please clear my doubt.


Q    When a page of a file is in page hash queue, does this page have page
table entry in any process ?
Q     Can this be discarded right away , if the need arises?


Nilesh Patel



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/