SysV swapped shared memory calculated incorrectly

SysV swapped shared memory calculated incorrectly

[Niko Jokinen]

Hi,

Tested on 2.6.28 and 2.6.31-rc4

SysV swapped shared memory is not calculated correctly
in /proc/<pid>/smaps and also by parsing /proc/<pid>/pagemap.
Rss value decreases also when swap is disabled, so this is where I am
lost as how shared memory is supposed to behave.

I have test program which makes 32MB shared memory segment and then I
use 'stress -m 1 --vm-bytes 120M', --vm-bytes is increased until rss
size decreases in smaps. Swap value never increases in smaps.

On the other hand shmctl(0, SHM_INFO, ...) does show shared memory in
swap because shm.c shm_get_stat() uses inodes to get values.


When test program is started:

shmctl() printout:
SHM_INFO (sys-wide):       total : 34580 kB
                             rss : 34388 kB
                            swap : 192 kB

smaps printout:
40153000-42153000 rw-s 00000000 00:08 1703949    /SYSV54016264 (deleted)
Size:              32768 kB
Rss:               32768 kB
Pss:               32768 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     32768 kB
Referenced:        32768 kB
Swap:                  0 kB

------------

After all memory is allocated (without swap smaps is the same, except
SHM_INFO shows 'Swap: 0' like it should), first byte is read hence the
4KB Referenced:

SHM_INFO (sys-wide):       total : 34580 kB
                             rss : 1528 kB
                            swap : 33052 kB


40153000-42153000 rw-s 00000000 00:08 1867789    /SYSV54016264 (deleted)
Size:              32768 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Swap:                  0 kB

------------

task_mmu.c, smaps_pte_range():

		if (is_swap_pte(ptent)) {
			mss->swap += PAGE_SIZE;
			continue;
		}

		if (!pte_present(ptent))
			continue;

When all memory is allocated pte_present() returns false for shared
memory. is_swap_pte() is never true for shared memory.

Ideas how to fix?


Br,
Niko Jokinen


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SysV swapped shared memory calculated incorrectly

[Hugh Dickins]

On Tue, 4 Aug 2009, Niko Jokinen wrote:
> 
> Tested on 2.6.28 and 2.6.31-rc4
> 
> SysV swapped shared memory is not calculated correctly
> in /proc/<pid>/smaps and also by parsing /proc/<pid>/pagemap.

smaps and pagemap are (reasonably) counting swap entries in the
page tables they're looking at.

But SysV shared memory is dealt with just like mmap of a tmpfs
file: we don't put swap entries into the page tables for that,
just as we don't put sector numbers into the page tables when
unmapping a diskfile page; the use of swapspace by that
filesystem is a lower-level detail not exposed at this level.

Well, we have had to expose "swap backed" near this level in
recent releases.  So it would be possible to recognize the
swap-backed shared vmas, and insert pte_file ptes instead
of pte_none ptes when unmapping pages from them, and adjust
the code which only expects those in nonlinear vmas, and
adjust smaps and pagemap to behave accordingly.

But I admit to having no appetite for any such change, cluttering
the main code just to touch up the anyhow rough picture that smaps
and pagemap are painting.  I much prefer to say that these areas are
backed by files, and it's a lower-level detail that those files are
backed by swap.

> Rss value decreases also when swap is disabled, so this is where I am
> lost as how shared memory is supposed to behave.

Did you check that detail on both 2.6.28 and 2.6.31-rc4?  I think
2.6.28 was unmapping the ptes from the pagetables, before the lower
level found that it had no swap to write them to; whereas a current
kernel didn't unmap them at all in my case.

> 
> I have test program which makes 32MB shared memory segment and then I
> use 'stress -m 1 --vm-bytes 120M', --vm-bytes is increased until rss
> size decreases in smaps. Swap value never increases in smaps.
> 
> On the other hand shmctl(0, SHM_INFO, ...) does show shared memory in
> swap because shm.c shm_get_stat() uses inodes to get values.

SHM-specific tools know they're dealing with tmpfs and perhaps swap,
and so can present a more tailored version of the info.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SysV swapped shared memory calculated incorrectly

[Niko Jokinen]

On Wed, 2009-08-05 at 20:21 +0200, ext Hugh Dickins wrote:
> On Tue, 4 Aug 2009, Niko Jokinen wrote:
> > 
> > Tested on 2.6.28 and 2.6.31-rc4
> > 
> > SysV swapped shared memory is not calculated correctly
> > in /proc/<pid>/smaps and also by parsing /proc/<pid>/pagemap.
> 
> smaps and pagemap are (reasonably) counting swap entries in the
> page tables they're looking at.
> 
> But SysV shared memory is dealt with just like mmap of a tmpfs
> file: we don't put swap entries into the page tables for that,
> just as we don't put sector numbers into the page tables when
> unmapping a diskfile page; the use of swapspace by that
> filesystem is a lower-level detail not exposed at this level.
> 
> Well, we have had to expose "swap backed" near this level in
> recent releases.  So it would be possible to recognize the
> swap-backed shared vmas, and insert pte_file ptes instead
> of pte_none ptes when unmapping pages from them, and adjust
> the code which only expects those in nonlinear vmas, and
> adjust smaps and pagemap to behave accordingly.
> 
> But I admit to having no appetite for any such change, cluttering
> the main code just to touch up the anyhow rough picture that smaps
> and pagemap are painting.  I much prefer to say that these areas are
> backed by files, and it's a lower-level detail that those files are
> backed by swap.
> 

This issue is originally from our performance team and they cannot
accurately measure per application memory usage if shared memory is
used. 
I guess workaround is to assume that following is true for shared memory
segments: Size-Rss = Swapped. (Since the issue below is fixed).

> > Rss value decreases also when swap is disabled, so this is where I am
> > lost as how shared memory is supposed to behave.
> 
> Did you check that detail on both 2.6.28 and 2.6.31-rc4?  I think
> 2.6.28 was unmapping the ptes from the pagetables, before the lower
> level found that it had no swap to write them to; whereas a current
> kernel didn't unmap them at all in my case.
> 

You are correct, tested on 2.6.31-rc5 and rss does not decrease anymore.

Br,
Niko Jokinen

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>