From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail144.messagelabs.com (mail144.messagelabs.com [216.82.254.51]) by kanga.kvack.org (Postfix) with SMTP id DC02B6B00EE for ; Tue, 19 Jul 2011 04:58:04 -0400 (EDT) References: <1309446135.48907.YahooMailNeo@web162020.mail.bf1.yahoo.com> <20110701065120.GA29530@suse.de> Message-ID: <1311065882.85318.YahooMailNeo@web162013.mail.bf1.yahoo.com> Date: Tue, 19 Jul 2011 01:58:02 -0700 (PDT) From: Pintu Agarwal Reply-To: Pintu Agarwal Subject: Re: JPEG image creation to display physical page state and memory fragmentation In-Reply-To: <20110701065120.GA29530@suse.de> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-709411929-1311065882=:85318" Sender: owner-linux-mm@kvack.org List-ID: To: "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" Cc: Mel Gorman --0-709411929-1311065882=:85318 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=0ADear All,=0A=A0=0AI wanted to represent the physical page state= of memory in the form of an image.=0AThe physical page state information s= uch as free pages, movable pages, reclaimable pages are taken from /proc/pa= getypeinfo.=0A(I know the buddyinfo and pagetypeinfo data may not be accura= te/insufficient but that is ok for me)=0A=A0=0AThese data will be used to c= reate a YUV image and then can be converted to a jpeg image.=0AThe image sh= ould clearly display the following(for each zone):-=0A- total free pages av= ailable --> in grey color=0A- total movable pages=A0=A0=A0=A0=A0=A0=A0 --> = in Red color- total=A0reclaimable pages=A0=A0=A0 --> in=A0Blue color=0AHere= each page will be represented by a pixel dot of different color.=0AA separ= ate image will be created for each zone.=0AI know how to create a raw YUV i= mage, but I am confused how to prepare an image layout based on the page-or= der and the total free page available.=0AIt could be as similar as the anti= -fragmentation page distribution image from below article from Mel Gorman.= =0Ahttp://lwn.net/Articles/224835/=0Ahttp://www.skynet.ie/~mel/anti-frag/20= 07-02-28/page_type_distribution.jpg=0ABut I am not able to understand how t= his image is formed.=0AFrom these image I am not able to find how many orde= r-n pages are free, movable and reclaimable.=0AIn the above image the block= size is 32x32 and the image layout is (32x32) x (32x32) =3D 1024x1024=0AAl= so each pixel represent one page in memory.=0ABut I am not able to understa= nd how this pixels are filled into the blocks.=0A=A0=0AI=A0wanted to create= an image to show memory fragmentation under each zone.=0AIf anybody has do= ne similar kind of work please let me know.=0A=A0=0AI have already=A0develo= ped a sample utility in linux to create=A0a dummy yuv420 image but could no= t able to figure out how to fill the pixels into the image.=0AMy yuv progra= m takes input as totalfreepages, totalmovablepages, totalreclaimablepages.= =0A=A0=0AMy system details are as folows:=0A- linux mobile running on ARM C= ortex-A9.=0A- total available RAM =3D 888MB=0A- Page size =3D 4K=0A- Page b= lock size =3D 1024 (from pagetypeinfo)=0A- page-order (0 to 10)=A0=A0=A0 (f= rom buddyinfo or pagetypeinfo)=0A=A0=0A=A0=0AThanks, Regards,=0APintu=0A=A0= =0A=0AFrom: Mel Gorman =0ATo: Pintu Agarwal =0ASent: Friday, 1 July 2011 12:21 PM=0ASubject: Re: JPEG image = creation for memory fragmentation=0A=0AOn Thu, Jun 30, 2011 at 08:02:15AM -= 0700, Pintu Agarwal wrote:=0A=0A> I need to create a sample jpeg image for = displaying page states=0A> and fragmentation info across each zone by colle= cting data from=0A> /proc/buddyinfo and /proc/pagetypeinfo a one instance o= f time.=0A>=0A> I saw your article about anti-fragmentation and page type d= istribution in the form of image from the following links:-=0A> http://lwn.= net/Articles/224835/=0A> http://www.skynet.ie/~mel/anti-frag/2007-02-28/pag= e_type_distribution.jpg=0A> =A0=0A> I wanted to understand, I to create the= similar image using my data=0A> from buddyinfo and pagetypeinfo in my case= .=0A=0AThere is insufficient data from buddyinfo and pagetypeinfo to genera= te=0Athe image linked above.=0A=0A> What logic you used for creating these = images?=0A=0AI used a perl script reading information via a kernel module a= nd=0Acreating the image with libgd. If I was doing it again, I would=0Aread= the necessary information from /proc/kpageflags. If I needed=0Aa greater d= egree of accuracy, I would patch the kernel to expand on=0Athe information = available in /proc/kpageflags.=0A=0A> How did you use the buddyinfo data to= form this image?=0A=0AI didn't use buddyinfo.=0A=0A> Can you give some det= ails about it?=0A> =A0=0A> Like your aticle I am also defining the followin= g:-=0A> - Each block size is 1024=0A> - Each page is one pixel=0A> - Free p= age will be gray color=0A> - Movable page will be green color=0A> - Reclaim= able page will be blue color=0A> - And (I dont want zoneboundary, percpu, p= inned pages etc in the image)=0A> =A0=0A> Now suppose my system page state = at some point of time looks like this:-=0A> -------------------------------= ----------------------------------------------------------=0A> Node:1, Zone= :DMA=0A> Order=A0=A0=A0 FreePages=A0=A0=A0=A0=A0=A0 MovablePages=A0=A0=A0 R= eclaimablePages=A0=A0=A0=A0=A0=A0=A0 Fragmentation[%]=0A> =A0 0=A0=A0=A0=A0= =A0=A0=A0=A0 38=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 36=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0%=0A> =A0 1=A0=A0=A0=A0=A0=A0=A0=A0 59= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 55=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 2=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0%=0A> =A0 2=A0=A0=A0=A0=A0=A0=A0=A0 40=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0 32=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 2%=0A> =A0 3=A0=A0=A0=A0=A0=A0=A0=A0 19=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0 16=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 4%=0A> =A0 = 4=A0=A0=A0=A0=A0=A0=A0=A0 12=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 10= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 6%=0A> =A0 5=A0=A0=A0=A0= =A0=A0=A0=A0=A0 4=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 2=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 9%=0A> =A0 6=A0=A0=A0=A0=A0=A0=A0= =A0=A0 6=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 4=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0 11%=0A> =A0 7=A0=A0=A0=A0=A0=A0=A0=A0=A0 7=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 4=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 16%=0A> =A0 8=A0=A0=A0=A0=A0=A0=A0=A0=A0 7=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 29%=0A> =A0 9=A0=A0=A0=A0=A0=A0=A0=A0=A0 2=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 55%=0A> = =A010=A0=A0=A0=A0=A0=A0=A0=A0=A0 2=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 2=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 70%=0A> TotalFreePag= es: 6932=0A> TotalMovablePages: 3954=0A> TotalReclaimablePages: 377=0A> Ove= rall Fragmentation: 18%=0A> -----------------------------------------------= ------------------------------------------=0A> How to represent this in the= form of a jpeg image?=0A=0AThere is insufficient data there to generate th= e same jpeg but I doubt=0Ait'd be very helpful anyway. Generating the image= s was a handy way of=0Avalidating anti-fragmentation was working as expecte= d and convincing=0Apeople that it was behaving in the right way but I never= used it to=0Aformally evaluate how well fragmentation was being handled. F= or that=0AI used unusable free space index, a fragmentation index, allocati= on=0Alatencies and success rates for allocating huge pages.=0A=0A-- =0AMel = Gorman=0ASUSE Labs --0-709411929-1311065882=:85318 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
       &nbs= p;        &nbs= p;          = ; 
Dear All,
 
I wanted to represen= t the physical page state of memory in the form of an image.
The physical page st= ate information such as free pages, movable pages, reclaimable pages are ta= ken from /proc/pagetypeinfo.
(I know the buddyinf= o and pagetypeinfo data may not be accurate/insufficient but that is ok for= me)
 
These data will be u= sed to create a YUV image and then can be converted to a jpeg image.=
The image should cle= arly display the following(for each zone):-
- total free pages a= vailable --> in grey color
- total movable page= s        --> in Red color
- total reclaim= able pages    --> in Blue color
Here each page will = be represented by a pixel dot of different color.
A separate image wil= l be created for each zone.
I know how to create= a raw YUV image, but I am confused how to prepare an image layout based on= the page-order and the total free page available.
It could be as simil= ar as the anti-fragmentation page distribution image from below article fro= m Mel Gorman.
http://lwn.ne= t/Articles/224835/
= http://www.skynet.ie/~mel/anti-frag/2007-02-28/page_type_distribution.jpg
But I am not able to understand how this image is formed.
From these image I a= m not able to find how many order-n pages are free, movable and reclaimable= .
In the above image t= he block size is 32x32 and the image layout is (32x32) x (32x32) =3D 1024x1= 024
Also each pixel repr= esent one page in memory.
But I am not able to= understand how this pixels are filled into the blocks.
 
I wanted to cre= ate an image to show memory fragmentation under each zone.
If anybody has done = similar kind of work please let me know.
 
I have already = developed a sample utility in linux to create a dummy yuv420 image but= could not able to figure out how to fill the pixels into the image.=
My yuv program takes= input as totalfreepages, totalmovablepages, totalreclaimablepages.<= /div>
 
My system details ar= e as folows:
- linux mobile running on ARM Cortex-A9.
- total available RA= M =3D 888MB
- Page size =3D 4K
- Page block size = =3D 1024 (from pagetypeinfo)
- page-order (0 to 1= 0)    (from buddyinfo or pagetypeinfo)
 
 
Thanks, Regards,
Pintu
 

From:= Mel Gorman <mgorman@suse.de>
To: Pintu Agarwal <pintu_agarwal@yahoo.com>Sent: Friday, 1 July 2011 = 12:21 PM
Subject: Re: JP= EG image creation for memory fragmentation

On Thu, Jun 30, 20= 11 at 08:02:15AM -0700, Pintu Agarwal wrote:

> I need to create a= sample jpeg image for displaying page states
> and fragmentation inf= o across each zone by collecting data from
> /proc/buddyinfo and /proc/pagetypeinfo a one instance of time.
>
> I saw your arti= cle about anti-fragmentation and page type distribution in the form of imag= e from the following links:-
> http://lwn.net/Articles/224835/=
> http://www.sk= ynet.ie/~mel/anti-frag/2007-02-28/page_type_distribution.jpg
> &n= bsp;
> I wanted to understand, I to create the similar image using my= data
> from buddyinfo and pagetypeinfo in my case.

There is i= nsufficient data from buddyinfo and pagetypeinfo to generate
the image l= inked above.

> What logic you used for creating these images?
=
I used a perl script reading information via a kernel module and
cre= ating the image with libgd. If I was doing it again, I would
read the necessary information from /proc/kpageflags. If I needed
a greater degr= ee of accuracy, I would patch the kernel to expand on
the information av= ailable in /proc/kpageflags.

> How did you use the buddyinfo data= to form this image?

I didn't use buddyinfo.

> Can you giv= e some details about it?
>  
> Like your aticle I am also = defining the following:-
> - Each block size is 1024
> - Each p= age is one pixel
> - Free page will be gray color
> - Movable p= age will be green color
> - Reclaimable page will be blue color
&g= t; - And (I dont want zoneboundary, percpu, pinned pages etc in the image)<= BR>>  
> Now suppose my system page state at some point of ti= me looks like this:-
> ----------------------------------------------= -------------------------------------------
> Node:1, Zone:DMA
>= ; Order    FreePages       MovablePages  &nbs= p; ReclaimablePages        Fragmentation= [%]
>   0         38&nbs= p;            &= nbsp; 36           &= nbsp;     1       &n= bsp;            = ;     0%
>   1     =     59         =       55       =           2   &= nbsp;           &nbs= p;         0%
>   2         40   &nbs= p;           32 &nbs= p;            &= nbsp;  1          &n= bsp;            = ;  2%
>   3        = 19            =    16          =        0      &= nbsp;           &nbs= p;      4%
>   4    = ;     12        = ;       10            =      1        &= nbsp;           &nbs= p;    6%
>   5      = ;    4         =        2      &= nbsp;          1  &n= bsp;            = ;          9%
>   6=           6   &= nbsp;            4&n= bsp;            = ;    1            &= nbsp;           11%
&g= t;   7          7 &n= bsp;            = ;  4           =       0       &= nbsp;           &nbs= p;    16%
>   8     &nbs= p;    7         = ;       0      =            1  &= nbsp;           &nbs= p;         29%
>   9          2 &nbs= p;            &= nbsp; 1           &n= bsp;     0       &nb= sp;            =     55%
>  10      =     2         &= nbsp;      2      &n= bsp;          0  &nb= sp;            =          70%
> TotalFreePages= : 6932
> TotalMovablePages: 3954
> TotalReclaimablePages: 377> Overall Fragmentation: 18%
> --------------------------------------------------------------------------= ---------------
> How to represent this in the form of a jpeg image?<= BR>
There is insufficient data there to generate the same jpeg but I dou= bt
it'd be very helpful anyway. Generating the images was a handy way of=
validating anti-fragmentation was working as expected and convincingpeople that it was behaving in the right way but I never used it to
for= mally evaluate how well fragmentation was being handled. For that
I used= unusable free space index, a fragmentation index, allocation
latencies = and success rates for allocating huge pages.

--
Mel Gorman
SU= SE Labs


--0-709411929-1311065882=:85318-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org