strategies for nicely handling large amount of uc/wc allocations.

strategies for nicely handling large amount of uc/wc allocations.

[Dave Airlie]

Hi linux-mm hackers,

So the situation I'm currently pondering has the following requirements.

1. Would like to re-use as much of shmfs as possible, it deals with
swap etc already.
2. Would like to allocate pages into shmfs vmas from a pool of pages
suitable for using in uc/wc mappings (without causing aliasing).
(either pages with no mappings or pages with their kernel mappings
already changed).
3. Would like the uc/wc page pool to grow/shrink with allocations and
memory pressure respectively.

An object allocated with this system can be in one of 3 states:

1. not in use by the GPU at all - pages mapped into user VMA, or even
swapped out.
2. mapped into the GPU GART - all pages locked into memory and all
mappings are uc/wc.
3. acting as backing store for a memory region in GPU VRAM - swap area
reserved, but may not need any pages in RAM,
any pages need to only have uc/wc mappings as to be migrated from VRAM
may involve mapping the backing store into the GART and blitting.

The issues I'm seeing with just doing my own pool of uncached pages
like the ia64 uncached allocator, is how do I deal with swapping
objects to disk and memory pressure.
considering GPU applications do a lot of memory allocations I can't
really just ignore swap.

I think ideally I'd like a zone for these sort of pages to become a
first class member of the VM, so that I can set a VMA to have an
uncached/wc bit,
then it sets a GFP uncached bit, and alloc_page goes and fetch
suitable pages from either the highmem zone, or from a resizeable
piece of normal zone.
This zone wouldn't be required to be a fixed size, and we could
migrate chunks of RAM into the zone as needed and flush them back out
under memory pressure. The big problem is we can't afford to migrate
pages at a time from the normal zone as the overhead is too much, so
it would have to be done in large blocks (probably around 1MB or so).

So I know this mail is probably a bit incoherent but I'm going around
in circles here on the best way to do this nicely, so I'm hoping
someone will either say this is crazy or point out something I've
missed.

Dave.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: strategies for nicely handling large amount of uc/wc allocations.

[Nick Piggin]

On Monday 23 June 2008 15:10, Dave Airlie wrote:

> The issues I'm seeing with just doing my own pool of uncached pages
> like the ia64 uncached allocator, is how do I deal with swapping
> objects to disk and memory pressure.

set_shrinker is the way to go for a first-pass approach.


> considering GPU applications do a lot of memory allocations I can't
> really just ignore swap.

It might be kind of nice to be able to use other things than
swap too. I've always wondered whether there is a problem with
having a userspace program allocate the memory and then register
it with the GPU allocator. This way you might be able to mmap a
file, or allocate anonymous memory or whatever you like. OTOH I
haven't thought through all the details this would require.


> I think ideally I'd like a zone for these sort of pages to become a
> first class member of the VM, so that I can set a VMA to have an
> uncached/wc bit,
> then it sets a GFP uncached bit, and alloc_page goes and fetch
> suitable pages from either the highmem zone, or from a resizeable
> piece of normal zone.

I doubt the performance requirements would justify putting them
into the allocator, and they don't seem to really fit the concept
of a zone.

Please start small and then we can see what works and what doesn't.
Start by not touching the VM at all, and just do what any other
driver would do and use .fault and/or ioctl+get_user_pages etc,
and set_shrinker to return pages back to the allocator.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: strategies for nicely handling large amount of uc/wc allocations.

[Dave Airlie]

On Mon, Jun 23, 2008 at 4:03 PM, Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> On Monday 23 June 2008 15:10, Dave Airlie wrote:
>
>> The issues I'm seeing with just doing my own pool of uncached pages
>> like the ia64 uncached allocator, is how do I deal with swapping
>> objects to disk and memory pressure.
>
> set_shrinker is the way to go for a first-pass approach.
>
>
>> considering GPU applications do a lot of memory allocations I can't
>> really just ignore swap.
>
> It might be kind of nice to be able to use other things than
> swap too. I've always wondered whether there is a problem with
> having a userspace program allocate the memory and then register
> it with the GPU allocator. This way you might be able to mmap a
> file, or allocate anonymous memory or whatever you like. OTOH I
> haven't thought through all the details this would require.

We've thought about that a few times, but really I'd rather avoid have
to have a userspace anything running
in order to suspend/resume. We are trying to move away from having
something like X controlling the world. Also a single program
brings with it a single program VMs space issues, we have GPUs with
1GB of RAM, thats a lot of backing store to have in one apps VM space
along with the objects in the GART.

>
>
>> I think ideally I'd like a zone for these sort of pages to become a
>> first class member of the VM, so that I can set a VMA to have an
>> uncached/wc bit,
>> then it sets a GFP uncached bit, and alloc_page goes and fetch
>> suitable pages from either the highmem zone, or from a resizeable
>> piece of normal zone.
>
> I doubt the performance requirements would justify putting them
> into the allocator, and they don't seem to really fit the concept
> of a zone.
>
> Please start small and then we can see what works and what doesn't.
> Start by not touching the VM at all, and just do what any other
> driver would do and use .fault and/or ioctl+get_user_pages etc,
> and set_shrinker to return pages back to the allocator.

But this loses the requirement of using shmfs, which one of the
current GPU driver is doing,
and what I'd like to have re-used. but maybe that just isn't worth it
at the moment.

We've actually done a few prototypes at this stage outside shmfs, and
the shmfs implementation (using cached pages) is quite neat.
It would be nice to try and go down the same road for the uncached
page required systems.

Dave.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: strategies for nicely handling large amount of uc/wc allocations.

[Nick Piggin]

On Monday 23 June 2008 16:18, Dave Airlie wrote:
> On Mon, Jun 23, 2008 at 4:03 PM, Nick Piggin <nickpiggin@yahoo.com.au> 
wrote:
> > On Monday 23 June 2008 15:10, Dave Airlie wrote:
> >> The issues I'm seeing with just doing my own pool of uncached pages
> >> like the ia64 uncached allocator, is how do I deal with swapping
> >> objects to disk and memory pressure.
> >
> > set_shrinker is the way to go for a first-pass approach.
> >
> >> considering GPU applications do a lot of memory allocations I can't
> >> really just ignore swap.
> >
> > It might be kind of nice to be able to use other things than
> > swap too. I've always wondered whether there is a problem with
> > having a userspace program allocate the memory and then register
> > it with the GPU allocator. This way you might be able to mmap a
> > file, or allocate anonymous memory or whatever you like. OTOH I
> > haven't thought through all the details this would require.
>
> We've thought about that a few times, but really I'd rather avoid have
> to have a userspace anything running
> in order to suspend/resume.

It wouldn't have to be running, it would just have to provide some
virtual memory area for get_user_pages to work on. Presumably does
not have to do anything when suspending and resuming?


> We are trying to move away from having 
> something like X controlling the world. Also a single program
> brings with it a single program VMs space issues, we have GPUs with
> 1GB of RAM, thats a lot of backing store to have in one apps VM space
> along with the objects in the GART.

I guess you could just ask for your own shmem inode from the kernel
and operate on that. It just seemed more flexible to pass in any
type of memory to register. But maybe it gets tricky as you have to
change the cache attributes.


> > Please start small and then we can see what works and what doesn't.
> > Start by not touching the VM at all, and just do what any other
> > driver would do and use .fault and/or ioctl+get_user_pages etc,
> > and set_shrinker to return pages back to the allocator.
>
> But this loses the requirement of using shmfs, which one of the
> current GPU driver is doing,
> and what I'd like to have re-used. but maybe that just isn't worth it
> at the moment.
>
> We've actually done a few prototypes at this stage outside shmfs, and
> the shmfs implementation (using cached pages) is quite neat.
> It would be nice to try and go down the same road for the uncached
> page required systems.

Well before you worry about swapping, you I guess you should at least
have an allocator that caches the UC pages and has a set_shrinker that
can be used to release unused pages under memory pressure? Do you have
that in yet?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>