persistent heap design advice

persistent heap design advice

[Keith Morgan]

I am interested in creating a persistent heap library and would
appreciate any suggestions on how to proceed. The 'persistent heap'
would be a region of virtual memory backed by a file and could be
expanded or contracted.

In order to build my 'persistent heap' it seems like I need a
fundamental facility that isn't provided by Linux. Please correct me if
I'm wrong! It would be something like mmap() ... but different. The
facility call it phmap for starters) would:

-map virtual addresses to a user-specified file
-coordinate the expansion/contraction of the file and the virtual
address space
-provide ram cache [of user-specified number of pages (cache itself is
nonpagable)]*
-provide load-on-demand of data from the file into the cache
-swap LRU pages back to the file when cache full

[]* I'm not sure if this is the right approach. I want to avoid paging
out user program/data when traversing very large 'persistent heaps'.

I an interested in writing at the highest possible level to create the
phmap facility. At this point my questions are very broad (I'm not
looking for a cookbook, just trying to prune the search space):

-Is is possible to hack the mmap() source to create it?
-If not, are there kernel/vm hooks that can be used to create it?
-If not these, how can it be done? (hopefully without hacking into the
kernel)

I've read the LDP documents on the memory architecture and Linux Device
Drivers is on its way from amazon.com. I am also starting to read the mm
and arch/../mm source but I must admit that I don't have a coherent
picture of memory management yet. Thanks for any insight.

Keith Morgan


--
To unsubscribe, send a message with 'unsubscribe linux-mm my@address'
in the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

Re: persistent heap design advice

[Ingo Oeser]

On Thu, 8 Apr 1999, Keith Morgan wrote:

> -If not, are there kernel/vm hooks that can be used to create it?
> kernel)

Look at 'include/linux/mm.h'.

There you can find 'struct vm_operations_struct', which provides
all the possible (and needed) hooks into the Linux-VMM.

Just take the implementation of 'mm/filemap.c' and modify it to
support your scheme of caching.

The only thing left to do is a SYSCALL-API-function. But this
isn't a real problem, is it?

Regards

Ingo Oeser
-- 
Feel the power of the penguin - run linux@your.pc
<esc>:x

--
To unsubscribe, send a message with 'unsubscribe linux-mm my@address'
in the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

Re: persistent heap design advice

[Eric W. Biederman]

>>>>> "KM" == Keith Morgan <kmorgan@inter-tax.com> writes:

KM> I am interested in creating a persistent heap library and would
KM> appreciate any suggestions on how to proceed. The 'persistent heap'
KM> would be a region of virtual memory backed by a file and could be
KM> expanded or contracted.

KM> In order to build my 'persistent heap' it seems like I need a
KM> fundamental facility that isn't provided by Linux. Please correct me if
KM> I'm wrong! It would be something like mmap() ... but different. The
KM> facility call it phmap for starters) would:

What do you see missing??
You obviously need a allactor built on top of your mmaped file but
besides that I don't see anything missing.

KM> -map virtual addresses to a user-specified file
mmap MAP_SHARED

KM> -coordinate the expansion/contraction of the file and the virtual
KM> address space
ftruncate, mmap, munmap

KM> -provide ram cache [of user-specified number of pages (cache itself is
KM> nonpagable)]*
why?? mlock

KM> -provide load-on-demand of data from the file into the cache
mmap MAP_SHARED

KM> -swap LRU pages back to the file when cache full
mmap MAP_SHARED


KM> []* I'm not sure if this is the right approach. I want to avoid paging
KM> out user program/data when traversing very large 'persistent heaps'.

Assuming a large persistent heap is larger than memory you can't help
but avoiding paging your heap.  Not paging other things much is to be desired of course.

An implementtion of madvise (so you can say which pages you aren't going to need for a while
and which pages you will be needing soon) is the only piece of the puzzle I see as
missing.

KM> I an interested in writing at the highest possible level to create the
KM> phmap facility. At this point my questions are very broad (I'm not
KM> looking for a cookbook, just trying to prune the search space):

Eric
--
To unsubscribe, send a message with 'unsubscribe linux-mm my@address'
in the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

Re: persistent heap design advice

[Kanoj Sarcar]

> 
> >>>>> "KM" == Keith Morgan <kmorgan@inter-tax.com> writes:
> 
> KM> I am interested in creating a persistent heap library and would
> KM> appreciate any suggestions on how to proceed. The 'persistent heap'
> KM> would be a region of virtual memory backed by a file and could be
> KM> expanded or contracted.
> 
> KM> In order to build my 'persistent heap' it seems like I need a
> KM> fundamental facility that isn't provided by Linux. Please correct me if
> KM> I'm wrong! It would be something like mmap() ... but different. The
> KM> facility call it phmap for starters) would:
> 
> What do you see missing??
> You obviously need a allactor built on top of your mmaped file but
> besides that I don't see anything missing.
> 
> KM> -map virtual addresses to a user-specified file
> mmap MAP_SHARED

And just be careful not to use the same file mmap'ed MAP_SHARED between
"unrelated" processes ...

> 
> KM> -coordinate the expansion/contraction of the file and the virtual
> KM> address space
> ftruncate, mmap, munmap

mremap might also come in handy, depending on how you want to handle
out-of-boundary requests ...

Kanoj
--
To unsubscribe, send a message with 'unsubscribe linux-mm my@address'
in the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/