http://ds9a.nl/cacheinfo project - please comment & improve

http://ds9a.nl/cacheinfo project - please comment & improve

[bert hubert]

Hello mm people!

I've written a module plus a tiny userspace program to query the page
cache. In short:

$ cinfo /lib/libc.so.6
/lib/libc.so.6: 182 of 272 (66.91%) pages in the cache, of which 0 (0.00%)
are dirty

Now, I'm a complete and utter beginner when it comes to kernelcoding. Also,
this is very much a 'release early, release often'-release. In other words,
it sucks & I know.

So I would like to ask you to look at it and send comments/patches to me.
I'm especially interested in architectural decisions - I currently export
data over a filesystem (cinfofs), which may or not be right.

The tarball (http://ds9a.nl/cacheinfo/cinfo-0.1.tar.gz) contains 2 manpages
which very lightly document how it works.

Thanks for your time!

Regards,

bert hubert

-- 
http://www.PowerDNS.com      Versatile DNS Services  
Trilab                       The Technology People   
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: http://ds9a.nl/cacheinfo project - please comment & improve

[Marcelo Tosatti]

On Sun, 27 May 2001, bert hubert wrote:

> Hello mm people!
> 
> I've written a module plus a tiny userspace program to query the page
> cache. In short:
> 
> $ cinfo /lib/libc.so.6
> /lib/libc.so.6: 182 of 272 (66.91%) pages in the cache, of which 0 (0.00%)
> are dirty
> 
> Now, I'm a complete and utter beginner when it comes to kernelcoding. Also,
> this is very much a 'release early, release often'-release. In other words,
> it sucks & I know.
> 
> So I would like to ask you to look at it and send comments/patches to me.
> I'm especially interested in architectural decisions - I currently export
> data over a filesystem (cinfofs), which may or not be right.
> 
> The tarball (http://ds9a.nl/cacheinfo/cinfo-0.1.tar.gz) contains 2 manpages
> which very lightly document how it works.

Hi Bert, 

You're using the "address_space->dirty_pages" list to calculate the number
of dirty pages.

Its interesting to note that pages on this list may not be really dirty
since we don't mark them clean when writting them out. (we only do that at
fdatasync/fsync time) 

So I suggest you to check for the PG_dirty (with the PageDirty macro) bit
on pages of that list to know if they are really dirty. 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: http://ds9a.nl/cacheinfo project - please comment & improve

[Marcelo Tosatti]

On Wed, 30 May 2001, bert hubert wrote:

> On Wed, May 30, 2001 at 04:54:12PM -0300, Marcelo Tosatti wrote:
> 
> > You're using the "address_space->dirty_pages" list to calculate the number
> > of dirty pages.
> 
> I was wondering about that. In limited testing I've never seen a non-0
> content of the dirty list. I ran:
> 
> dd if=/dev/zero of=test count=100000 &
> while true ; do ./cinfo test; done
> 
> And saw no dirty pages. 

Oops.

You will see no dirty pages here anyway --- data written through
write() is commited to the buffer cache directly. 

You can loop in each page into the clean_list and check their
"page->buffers" pointer.

If there are dirty buffer_head's there, you can count the page as dirty. 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: http://ds9a.nl/cacheinfo project - please comment & improve

[bert hubert]

On Wed, May 30, 2001 at 04:54:12PM -0300, Marcelo Tosatti wrote:

> You're using the "address_space->dirty_pages" list to calculate the number
> of dirty pages.

I was wondering about that. In limited testing I've never seen a non-0
content of the dirty list. I ran:

dd if=/dev/zero of=test count=100000 &
while true ; do ./cinfo test; done

And saw no dirty pages. 

> So I suggest you to check for the PG_dirty (with the PageDirty macro) bit
> on pages of that list to know if they are really dirty. 

Ok - will do. I plan to release a slightly improved version shortly that
addresses this issue. Thanks!

Oh, if anybody has ideas on statistics that should be exported, please let
me know. On the agenda is a bitmap that describes which pages are actually
in the cache.

Regards,

bert

-- 
http://www.PowerDNS.com      Versatile DNS Services  
Trilab                       The Technology People   
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: http://ds9a.nl/cacheinfo project - please comment & improve

[Ingo Oeser]

On Wed, May 30, 2001 at 11:48:06PM +0200, bert hubert wrote:
> Oh, if anybody has ideas on statistics that should be exported, please let
> me know. On the agenda is a bitmap that describes which pages are actually
> in the cache.

You mean sth. like the mincore() syscall?

Regards

Ingo Oeser
-- 
To the systems programmer,
users and applications serve only to provide a test load.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: http://ds9a.nl/cacheinfo project - please comment & improve

[bert hubert]

On Thu, May 31, 2001 at 07:17:30PM +0200, Ingo Oeser wrote:
> On Wed, May 30, 2001 at 11:48:06PM +0200, bert hubert wrote:
> > Oh, if anybody has ideas on statistics that should be exported, please let
> > me know. On the agenda is a bitmap that describes which pages are actually
> > in the cache.
> 
> You mean sth. like the mincore() syscall?

If you first mmap() the file that would probably work. In dire need of a
manpage though - I'll whip one up and send it to Andries. Probably explains
its relative lack of popularity - I'd never heard of mincore() although it's
been around since BSD4.4 it appears.

Pretty sad that it wastes 7 bits per byte though, but standards conformance
is also useful.

Regards,

bert

-- 
http://www.PowerDNS.com      Versatile DNS Services  
Trilab                       The Technology People   
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: http://ds9a.nl/cacheinfo project - please comment & improve

[bert hubert]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

On Thu, May 31, 2001 at 11:53:27PM +0200, bert hubert wrote:
> > > Oh, if anybody has ideas on statistics that should be exported, please let
> > > me know. On the agenda is a bitmap that describes which pages are actually
> > > in the cache.
> > 
> > You mean sth. like the mincore() syscall?
> 
> If you first mmap() the file that would probably work. In dire need of a
> manpage though - I'll whip one up and send it to Andries. Probably explains
> its relative lack of popularity - I'd never heard of mincore() although it's
> been around since BSD4.4 it appears.

As promised, a manpage. I alreasy sent it to Andries but the people over
here may also have comments.

Regards,

bert

-- 
http://www.PowerDNS.com      Versatile DNS Services  
Trilab                       The Technology People   
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

[-- Attachment #2: mincore.2 --]
[-- Type: text/plain, Size: 3228 bytes --]

.\" Hey Emacs! This file is -*- nroff -*- source.
.\"
.\" Copyright (C) 2001 Andries Brouwer (aeb@cwi.nl)
.\"
.\" Permission is granted to make and distribute verbatim copies of this
.\" manual provided the copyright notice and this permission notice are
.\" preserved on all copies.
.\"
.\" Permission is granted to copy and distribute modified versions of this
.\" manual under the conditions for verbatim copying, provided that the
.\" entire resulting derived work is distributed under the terms of a
.\" permission notice identical to this one
.\" 
.\" Since the Linux kernel and libraries are constantly changing, this
.\" manual page may be incorrect or out-of-date.  The author(s) assume no
.\" responsibility for errors or omissions, or for damages resulting from
.\" the use of the information contained herein.  The author(s) may not
.\" have taken the same level of care in the production of this manual,
.\" which is licensed free of charge, as they might when working
.\" professionally.
.\" 
.\" Formatted or processed versions of this manual, if unaccompanied by
.\" the source, must acknowledge the copyright and authors of this work.
.\"
.\" Created Sun Jun 3 17:23:32 2001 by bert hubert <ahu@ds9a.nl>
.\"
.TH MINCORE 2 "3 June 2001" "Linux 2.4.5" "Linux Programmer's Manual"
.SH NAME
mincore \- get information on whether pages are in core
.SH SYNOPSIS
.B #include <unistd.h>
.br
.B #include <sys/mman.h>
.sp
.BI "int mincore(void *" start ", size_t " length ", unsigned char * " vec );
.SH DESCRIPTION
The
.B mincore
function requests a vector describing which pages of a file are in core and
can be read without disk access. The kernel will supply data for
.I length
bytes following the 
.I start
address. On return, the kernel will have filled
.I vec
with bytes, of which the least significant bit indicates if a page is 
core resident.

For
.B mincore
to return succesfully, 
.I start
must lie on a page boundary. It is the caller's responsibility to round up to the nearest page. The
.I length
parameter need not be a multiple of the page size. The vector
.I vec
must be large enough to contain length/PAGE_SIZE bytes.

.SH "RETURN VALUE"
On success,
.B mincore
returns zero.
On error, \-1 is returned, and
.I errno
is set appropriately.
.SH ERRORS
.B EAGAIN
kernel is temporarily out of resources
.TP
.B EINVAL
.i start
is not a multiple of PAGE_CACHE_SIZE (PAGE_SIZE) or 
.i len
has a non-positive value
.TP
.B EFAULT
.I vec
points to an illegal address
.TP
.B ENOMEM
.I address
to
.I address
+
.I length
contained unmapped memory, or memory not part of a file.

.SH "BUGS"
.B mincore
should return a bit vector and not a byte vector. As of Linux 2.4.5, it is not
possible to gain information on the core residency of pages which are not backed by a file. 
In other words, calling 
.B mincore
on an region returned by an anonymous
.B mmap(2)
does not work and sets errno to ENOMEM. Unless pages are locked in memory, the contents of
.I vec
may be stale by the time they reach userspace.

.SH "CONFORMING TO"
.B mincore
does not appear to be part of POSIX or the Single Unix Specification. 
.SH HISTORY
The mincore() function first appeared in 4.4BSD

.SH "SEE ALSO"
.BR getpagesize (2),
.BR mmap (2)

Re: http://ds9a.nl/cacheinfo project - please comment & improve

[Hugh Dickins]

On Sun, 3 Jun 2001, bert hubert wrote:
> 
> As promised, a manpage. I alreasy sent it to Andries but the people over
> here may also have comments.

I think cut out the reference to PAGE_CACHE_SIZE.  mincore()'s current
kernel implementation may indeed involve PAGE_CACHE_SIZE, but I believe
that if PAGE_CACHE_SIZE is ever changed away from PAGE_SIZE, that will
be a kernel implementation detail, which should not affect its interfaces
with user-space.  You're right to SEE ALSO getpagesize: really even the
references to PAGE_SIZE should defer to getpagesize e.g. the mmap man
page (on my RH7 anyway) says "offset should ordinarily be a multiple of
the page size returned by getpagesize(2)"; but it may get tiresome to use
that circumlocution ("the artist formerly known as Prince") everywhere.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/