RFC: swaptrace tool

RFC: swaptrace tool

[Peter Teoh]

Go through this:

http://linux-mm.org/LinuxMMProjects

and u find there is no swaptrace.   What I want is a visualization of
how the swap is being use.   So once the operation is started, all
swap operation will be immediately written to an area in memory,
showing how the swap is written - the destination begin offset /
destination end offset/size info, and by what process/task - and its
correspond source begin offset, and source end offset.   The data
content itself will not be recorded.   Then after some time, via
ioctl() control, it will be stopped, and all that have been written to
memory will be flushed out to a file.   This flushing to external file
only take place after the data collection has stopped, otherwise, the
swap operations itself will affect the behavior of the swap, thus
rendering the data collection invalid.

The purpose of this trace is to see/oberver how the swap is being
used, whether any algor can help to cluster the swap together so as to
enhance swap batch processing etc.

Any comment on this idea?   Will it be useful?

-- 
Regards,
Peter Teoh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: swaptrace tool

[Peter Chubb]

>>>>> "Peter" == Peter Teoh <htmldeveloper@gmail.com> writes:

Peter> Go through this: http://linux-mm.org/LinuxMMProjects

Peter> and u find there is no swaptrace.  What I want is a
Peter> visualization of how the swap is being use.  So once the
Peter> operation is started, all swap operation will be immediately
Peter> written to an area in memory, showing how the swap is written -
Peter> the destination begin offset / destination end offset/size
Peter> info, and by what process/task - and its correspond source
Peter> begin offset, and source end offset.  The data content itself
Peter> will not be recorded.  Then after some time, via ioctl()
Peter> control, it will be stopped, and all that have been written to
Peter> memory will be flushed out to a file.  This flushing to
Peter> external file only take place after the data collection has
Peter> stopped, otherwise, the swap operations itself will affect the
Peter> behavior of the swap, thus rendering the data collection
Peter> invalid.

You could probably do this with blktrace for a swap partition, which
can do exactly this on any block device.   For swap files, it's a bit harder.

--
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
http://www.ertos.nicta.com.au           ERTOS within National ICT Australia

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: swaptrace tool

[Mulyadi Santosa]

hi

On Tue, Apr 1, 2008 at 3:54 PM, Peter Teoh <htmldeveloper@gmail.com> wrote:
> Go through this:
>
>  http://linux-mm.org/LinuxMMProjects

once I posted unfinished per process swap usage statistics to
LKML....IIRC around mid 2007. Somebody care to look (again) and helped
me to improve it?

regards,

Mulyadi.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: swaptrace tool

[KOSAKI Motohiro]

hi

> once I posted unfinished per process swap usage statistics to
> LKML....IIRC around mid 2007. Somebody care to look (again) and helped
> me to improve it?

hmm,
I can't find it from mailing list log.
Could you recall subject of that patch?



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: swaptrace tool

[Mulyadi Santosa]

Hello..

On Tue, Apr 1, 2008 at 7:27 PM, KOSAKI Motohiro
<kosaki.motohiro@jp.fujitsu.com> wrote:
> hi
>
>
>  > once I posted unfinished per process swap usage statistics to
>  > LKML....IIRC around mid 2007. Somebody care to look (again) and helped
>  > me to improve it?
>
>  hmm,
>  I can't find it from mailing list log.
>  Could you recall subject of that patch?

here is the URL http://lkml.org/lkml/2006/8/4/77

regards,

Mulyadi.

--
To unsubscribe from this list: send an email with
"unsubscribe kernelnewbies" to ecartis@nl.linux.org
Please read the FAQ at http://kernelnewbies.org/FAQ

Re: RFC: swaptrace tool

[KOSAKI Motohiro]

Hi

> >  hmm,
> >  I can't find it from mailing list log.
> >  Could you recall subject of that patch?
> 
> here is the URL http://lkml.org/lkml/2006/8/4/77

Oh! thanks.
but, this patch only get per process # of swap pages.

and the following patch is merged -mm before a while.
http://marc.info/?l=linux-kernel&m=120654533828554&w=2

thus, now we have its capability via /proc/pid/smap.

in my understanding, originally requirement wanted following tuple.

  - destination begin offset / size
  - source begin offset / size
  - process/task


Thanks.



--
To unsubscribe from this list: send an email with
"unsubscribe kernelnewbies" to ecartis@nl.linux.org
Please read the FAQ at http://kernelnewbies.org/FAQ

Re: RFC: swaptrace tool

[Peter Teoh]

On Wed, Apr 2, 2008 at 10:58 AM, KOSAKI Motohiro
<kosaki.motohiro@jp.fujitsu.com> wrote:
> Hi
>
>
>  > >  hmm,
>  > >  I can't find it from mailing list log.
>  > >  Could you recall subject of that patch?
>  >
>  > here is the URL http://lkml.org/lkml/2006/8/4/77
>
>  Oh! thanks.
>  but, this patch only get per process # of swap pages.
>
>  and the following patch is merged -mm before a while.
>  http://marc.info/?l=linux-kernel&m=120654533828554&w=2
>
>  thus, now we have its capability via /proc/pid/smap.
>
>  in my understanding, originally requirement wanted following tuple.
>
>   - destination begin offset / size
>   - source begin offset / size
>   - process/task
>
>

ah...yes...exactly.   but Mulyadi patch is good headstart for me to
focus on - to see how it can be tweaked for extracting out this lean
information.   lean because as compared with blktrace, it is just
focusing on the BOUNDARY of the swapspace being written or read
(whereas blktrace will include the data itself).....every seconds or
every minutes, snapshot of these kind of information will allow us to
see how the swapspace is dynamically being used....and then we can
then design a good page replacement policy or swapspace usage
algorithm.....eg, to maximize the clustering factor when paging.
sort of a simulation tool......

-- 
Regards,
Peter Teoh

--
To unsubscribe from this list: send an email with
"unsubscribe kernelnewbies" to ecartis@nl.linux.org
Please read the FAQ at http://kernelnewbies.org/FAQ

Re: RFC: swaptrace tool

[Mulyadi Santosa]

Hi Scott...

On Wed, Apr 2, 2008 at 10:47 AM, Scott Lovenberg
<scott.lovenberg@gmail.com> wrote:
>  Are you basing it on Least Recently Used?  I seems to remember that Con
> Konilias had added a swapping module to his staircase scheduler...

yes, Con named it swap prefetch but it doesn't tightly related to
Staircase scheduler.

>if I
> remember correctly, it was fairly controversial because it would swap back
> in before the data was needed if the cache pressure was low and the memory
> was freed back up; although I'm not sure if you want pursue paying the swap
> price twice, it seems to make sense that the cost doesn't count if the
> machine is idle anyways and you don't want to wait for a double page fault
> (first for the page table and then for the data itself) when you're
> requesting data.

AFAIK, page table in Linux is never swapped out (they reside in kernel
space and kernel space is locked in RAM all the time). In windows,
maybe...

regards,

Mulyadi.

--
To unsubscribe from this list: send an email with
"unsubscribe kernelnewbies" to ecartis@nl.linux.org
Please read the FAQ at http://kernelnewbies.org/FAQ

Re: RFC: swaptrace tool

[Peter Teoh]

On Wed, Apr 2, 2008 at 1:49 PM, Mulyadi Santosa
<mulyadi.santosa@gmail.com> wrote:
> Hi all
>
>  On Wed, Apr 2, 2008 at 9:58 AM, KOSAKI Motohiro
>
> <kosaki.motohiro@jp.fujitsu.com> wrote:
>
> > Hi
>  >
>  >
>  >  > >  hmm,
>  >  > >  I can't find it from mailing list log.
>  >  > >  Could you recall subject of that patch?
>  >  >
>  >  > here is the URL http://lkml.org/lkml/2006/8/4/77
>  >
>  >  Oh! thanks.
>  >  but, this patch only get per process # of swap pages.
>
>  yeah, I was newbie and still newbie until this very moment :)
>
>
>  >  and the following patch is merged -mm before a while.
>  >  http://marc.info/?l=linux-kernel&m=120654533828554&w=2
>
>  arghhh, peterz beat me again! :D


Did he???

>
>
>  >  thus, now we have its capability via /proc/pid/smap.
>

For a criticism of smaps:

http://bmaurer.blogspot.com/2006/03/memory-usage-with-smaps.html

and btw....i cannot find any "swp" or "swap" output.   so peterz and
yours are non-overlapping.

-- 
Regards,
Peter Teoh

--
To unsubscribe from this list: send an email with
"unsubscribe kernelnewbies" to ecartis@nl.linux.org
Please read the FAQ at http://kernelnewbies.org/FAQ

Re: RFC: swaptrace tool

[KOSAKI Motohiro]

Hi

> >  >  thus, now we have its capability via /proc/pid/smap.
> 
> For a criticism of smaps:
> 
> http://bmaurer.blogspot.com/2006/03/memory-usage-with-smaps.html
> 
> and btw....i cannot find any "swp" or "swap" output.   so peterz and
> yours are non-overlapping.

please looks my mail again.
Peter Zijlstra added swap output feature.

  ->  http://marc.info/?l=linux-kernel&m=120654533828554&w=2



--
To unsubscribe from this list: send an email with
"unsubscribe kernelnewbies" to ecartis@nl.linux.org
Please read the FAQ at http://kernelnewbies.org/FAQ

Re: RFC: swaptrace tool

[Peter Teoh]

On Wed, Apr 2, 2008 at 3:55 PM, KOSAKI Motohiro
<kosaki.motohiro@jp.fujitsu.com> wrote:

>  please looks my mail again.
>  Peter Zijlstra added swap output feature.
>
>   ->  http://marc.info/?l=linux-kernel&m=120654533828554&w=2
>
>
>
Oops...sorry, I never see the date....it was dated 26 Mar 2008,
perhaps still in -mm, my latest 2.6.25-rc7 (linus git tree) never got
the "Swap" yet :-).   Thanks.

-- 
Regards,
Peter Teoh

--
To unsubscribe from this list: send an email with
"unsubscribe kernelnewbies" to ecartis@nl.linux.org
Please read the FAQ at http://kernelnewbies.org/FAQ