[PATCH 0/6] CKRM: Memory controller for CKRM

[PATCH 0/6] CKRM: Memory controller for CKRM

[Chandra Seetharaman]

Hello ckrm-tech members,

Here is the latest CKRM Memory controller patch against the patchset Gerrit
released on 05/05/05.

I applied the feedback I got on/off the list. Made few fixes and some
cleanups. Details about the changes are in the appripriate patches.

It is tested on i386.

Currently disabled on NUMA.

Hello linux-mm members,

These are set of patches that provides the control of memory under the CKRM
framework(Details at http://ckrm.sf.net). I eagerly wait for your
feedback/comments/suggestions/concerns etc.,

To All,

I am looking for improvement suggestions
        - to not have a field in the page data structure for the mem
          controller
	- to make vmscan.c cleaner.

--------
Patches are
11-01-mem_base_changes:
        Basic changes to the core kernel to support memory controller.

11-02-mem_base-core:
        To fit in the ckrm framework. No support for guarantee, limit in
	this patch. No config file support also.

11-03-mem_core-limit:
        Support for limit is added.

11-04-mem_limit-guar:
        Support for guarantee is added.

11-05-mem_guar-config:
        Support for few config parameters added.

11-06-mem_config-docs:
        Ofcourse... Documentation.

regards,

chandra


-- 

----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - sekharan@us.ibm.com   |      .......you may get it.
----------------------------------------------------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH 0/6] CKRM: Memory controller for CKRM

[Hirokazu Takahashi]

Hello,

It just looks like that once kswapd moves pages between the active lists
and the inactive lists, the pages happen to belong to the class
to which kswapd belong.

Is this right behavior that you intend?

> Hello ckrm-tech members,
> 
> Here is the latest CKRM Memory controller patch against the patchset Gerrit
> released on 05/05/05.
> 
> I applied the feedback I got on/off the list. Made few fixes and some
> cleanups. Details about the changes are in the appripriate patches.
> 
> It is tested on i386.
> 
> Currently disabled on NUMA.
> 
> Hello linux-mm members,
> 
> These are set of patches that provides the control of memory under the CKRM
> framework(Details at http://ckrm.sf.net). I eagerly wait for your
> feedback/comments/suggestions/concerns etc.,
> 
> To All,
> 
> I am looking for improvement suggestions
>         - to not have a field in the page data structure for the mem
>           controller

What do you think if you make each class owns inodes instead of pages
in the page-cache?

> 	- to make vmscan.c cleaner.


Thanks,
Hirokazu Takahashi.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH 0/6] CKRM: Memory controller for CKRM

[Chandra Seetharaman]

On Thu, May 19, 2005 at 10:43:25AM +0900, Hirokazu Takahashi wrote:
> Hello,
> 
> It just looks like that once kswapd moves pages between the active lists
> and the inactive lists, the pages happen to belong to the class
> to which kswapd belong.

In refill_inactive_zone()(where pages are moved from active to inactive
list), ckrm_zone(where the page came from) is where the inactive pages are 
moved to.

I don't see how you concluded this. Can you point to the code.

> 
> Is this right behavior that you intend?

certainly not :)
> 
> > Hello ckrm-tech members,
> > 
> > Here is the latest CKRM Memory controller patch against the patchset Gerrit
> > released on 05/05/05.
> > 
> > I applied the feedback I got on/off the list. Made few fixes and some
> > cleanups. Details about the changes are in the appripriate patches.
> > 
> > It is tested on i386.
> > 
> > Currently disabled on NUMA.
> > 
> > Hello linux-mm members,
> > 
> > These are set of patches that provides the control of memory under the CKRM
> > framework(Details at http://ckrm.sf.net). I eagerly wait for your
> > feedback/comments/suggestions/concerns etc.,
> > 
> > To All,
> > 
> > I am looking for improvement suggestions
> >         - to not have a field in the page data structure for the mem
> >           controller
> 
> What do you think if you make each class owns inodes instead of pages
> in the page-cache?
> 
> > 	- to make vmscan.c cleaner.
> 
> 
> Thanks,
> Hirokazu Takahashi.
> 
> 
> 

-- 

----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - sekharan@us.ibm.com   |      .......you may get it.
----------------------------------------------------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH 0/6] CKRM: Memory controller for CKRM

[Hirokazu Takahashi]

Hi Chandra,

> On Thu, May 19, 2005 at 10:43:25AM +0900, Hirokazu Takahashi wrote:
> > Hello,
> > 
> > It just looks like that once kswapd moves pages between the active lists
> > and the inactive lists, the pages happen to belong to the class
> > to which kswapd belong.
> 
> In refill_inactive_zone()(where pages are moved from active to inactive
> list), ckrm_zone(where the page came from) is where the inactive pages are 
> moved to.

Ah, I understood.
You have changed these functions not to call add_page_to_active_list() or
add_page_to_inactive_list() anymore.

Still, there may remain problems that mark_page_accessed() calls
add_page_to_active_list() to move pages between classes.
I guess this isn't good manner since some functions which call
mark_page_accessed(), like unmap_mapping_range_vma() or get_user_pages(),
may refer pages of the other classes.

> I don't see how you concluded this. Can you point to the code.
> 
> > 
> > Is this right behavior that you intend?
> 
> certainly not :)
> > 
> > > Hello ckrm-tech members,
> > > 
> > > Here is the latest CKRM Memory controller patch against the patchset Gerrit
> > > released on 05/05/05.
> > > 
> > > I applied the feedback I got on/off the list. Made few fixes and some
> > > cleanups. Details about the changes are in the appripriate patches.
> > > 
> > > It is tested on i386.
> > > 
> > > Currently disabled on NUMA.
> > > 
> > > Hello linux-mm members,
> > > 
> > > These are set of patches that provides the control of memory under the CKRM
> > > framework(Details at http://ckrm.sf.net). I eagerly wait for your
> > > feedback/comments/suggestions/concerns etc.,
> > > 
> > > To All,
> > > 
> > > I am looking for improvement suggestions
> > >         - to not have a field in the page data structure for the mem
> > >           controller
> > 
> > What do you think if you make each class owns inodes instead of pages
> > in the page-cache?
> > 
> > > 	- to make vmscan.c cleaner.


Thanks,
Hirokazu Takahashi.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH 0/6] CKRM: Memory controller for CKRM

[Chandra Seetharaman]

On Fri, May 20, 2005 at 02:29:27PM +0900, Hirokazu Takahashi wrote:
> Hi Chandra,
> 
> > On Thu, May 19, 2005 at 10:43:25AM +0900, Hirokazu Takahashi wrote:
> > > Hello,
> > > 
> > > It just looks like that once kswapd moves pages between the active lists
> > > and the inactive lists, the pages happen to belong to the class
> > > to which kswapd belong.
> > 
> > In refill_inactive_zone()(where pages are moved from active to inactive
> > list), ckrm_zone(where the page came from) is where the inactive pages are 
> > moved to.
> 
> Ah, I understood.
> You have changed these functions not to call add_page_to_active_list() or
> add_page_to_inactive_list() anymore.
> 
> Still, there may remain problems that mark_page_accessed() calls
> add_page_to_active_list() to move pages between classes.
> I guess this isn't good manner since some functions which call
> mark_page_accessed(), like unmap_mapping_range_vma() or get_user_pages(),
> may refer pages of the other classes.

You mean these functions are not called in the context of the task that
is in the stack ?

> 
<snip>
> > > > 
> > > > I am looking for improvement suggestions
> > > >         - to not have a field in the page data structure for the mem
> > > >           controller
> > > 
> > > What do you think if you make each class owns inodes instead of pages
> > > in the page-cache?

I think i missed to answer this question in the earlier reply.

do you mean a controller for managing inodes ?
> > > 
> > > > 	- to make vmscan.c cleaner.
> 
> 
> Thanks,
> Hirokazu Takahashi.

-- 

----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - sekharan@us.ibm.com   |      .......you may get it.
----------------------------------------------------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH 0/6] CKRM: Memory controller for CKRM

[Hirokazu Takahashi]

Hi Chandra,

> > > On Thu, May 19, 2005 at 10:43:25AM +0900, Hirokazu Takahashi wrote:
> > > > Hello,
> > > > 
> > > > It just looks like that once kswapd moves pages between the active lists
> > > > and the inactive lists, the pages happen to belong to the class
> > > > to which kswapd belong.
> > > 
> > > In refill_inactive_zone()(where pages are moved from active to inactive
> > > list), ckrm_zone(where the page came from) is where the inactive pages are 
> > > moved to.
> > 
> > Ah, I understood.
> > You have changed these functions not to call add_page_to_active_list() or
> > add_page_to_inactive_list() anymore.
> > 
> > Still, there may remain problems that mark_page_accessed() calls
> > add_page_to_active_list() to move pages between classes.
> > I guess this isn't good manner since some functions which call
> > mark_page_accessed(), like unmap_mapping_range_vma() or get_user_pages(),
> > may refer pages of the other classes.
> 
> You mean these functions are not called in the context of the task that
> is in the stack ?

No, in some cases though it doesn't seem to be serious.

You may take a look at unmap_mapping_range_tree() as an example.
It traverses mapping->i_mmap to find all related vma's to be
unmapped. And get_user_pages() has the parameter that indicates
which process space should be accessed.

Thanks,
Hirokazu Takahashi.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH 0/6] CKRM: Memory controller for CKRM

[Hirokazu Takahashi]

Hi Chandra,

> > > > > 
> > > > > I am looking for improvement suggestions
> > > > >         - to not have a field in the page data structure for the mem
> > > > >           controller
> > > > 
> > > > What do you think if you make each class owns inodes instead of pages
> > > > in the page-cache?
> 
> I think i missed to answer this question in the earlier reply.
> 
> do you mean a controller for managing inodes ?

Yes, I think it might be another solution though I haven't examined
about it yet.


Thanks,
Hirokazu Takahashi.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH 0/6] CKRM: Memory controller for CKRM

[Hirokazu Takahashi]

Hi Chandra,

> Here is the latest CKRM Memory controller patch against the patchset Gerrit
> released on 05/05/05.
    (snip)
> These are set of patches that provides the control of memory under the CKRM
> framework(Details at http://ckrm.sf.net). I eagerly wait for your
> feedback/comments/suggestions/concerns etc.,
> 
> To All,
> 
> I am looking for improvement suggestions
>         - to not have a field in the page data structure for the mem
>           controller
> 	- to make vmscan.c cleaner.

I think it's very heavy to move all pages, which are mapped to removing
regions, to new classes every time. It always happens when doing exec()
or exit(), while munmap and closing file don't.
Pages associating with libc.so or text of shells might move around
the all classes.

IMHO, it would be enough to just leave them as they are.
These pages would be released a little later if no class touch them,
or they might be accessed from another class to migrate another
class, or they might reused in the same class.

I feel it's not needed to move these pages in hurry.
I prefer the implementation light.
What do you think?


BTW, the memory controller would be a good new to video streaming
guys, I guess.


Thanks,
Hirokazu Takahashi.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH 0/6] CKRM: Memory controller for CKRM

[Chandra Seetharaman]

On Fri, May 20, 2005 at 06:26:24PM +0900, Hirokazu Takahashi wrote:
> Hi Chandra,
> 
> 
> I think it's very heavy to move all pages, which are mapped to removing
> regions, to new classes every time. It always happens when doing exec()
> or exit(), while munmap and closing file don't.
> Pages associating with libc.so or text of shells might move around
> the all classes.
> 
> IMHO, it would be enough to just leave them as they are.
> These pages would be released a little later if no class touch them,
> or they might be accessed from another class to migrate another
> class, or they might reused in the same class.

No, it will be incorrect, as the class that is using the page doesn't need
these pages anymore, we should not be charging them for those pages.

May be we should do something light while keeping the accounting proper.
Any ideas ?
> 
> I feel it's not needed to move these pages in hurry.
> I prefer the implementation light.
> What do you think?
> 
> 
> BTW, the memory controller would be a good new to video streaming
> guys, I guess.
> 
> 
> Thanks,
> Hirokazu Takahashi.
> 

-- 

----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - sekharan@us.ibm.com   |      .......you may get it.
----------------------------------------------------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH 0/6] CKRM: Memory controller for CKRM

[Hirokazu Takahashi]

Hi Chandra,

> > I think it's very heavy to move all pages, which are mapped to removing
> > regions, to new classes every time. It always happens when doing exec()
> > or exit(), while munmap and closing file don't.
> > Pages associating with libc.so or text of shells might move around
> > the all classes.
> > 
> > IMHO, it would be enough to just leave them as they are.
> > These pages would be released a little later if no class touch them,
> > or they might be accessed from another class to migrate another
> > class, or they might reused in the same class.
> 
> No, it will be incorrect, as the class that is using the page doesn't need
> these pages anymore, we should not be charging them for those pages.
> 
> May be we should do something light while keeping the accounting proper.
> Any ideas ?

I understand your intention. But I don't think it's always proper.
It wouldn't be bad idea to leave pages for perl scripts launched
by Apache, as they may be reused soon.
Each class may tend to re-execute the same programs.

OTOH, pages as a cache of closed file may not be reused by the same
class again, though they're counted as valid pages for the class.

I guess both of them are in gray area.

> > I feel it's not needed to move these pages in hurry.
> > I prefer the implementation light.
> > What do you think?
> > 
> > 
> > BTW, the memory controller would be a good new to video streaming
> > guys, I guess.


Thanks,
Hirokazu Takahashi.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Virtual NUMA machine and CKRM

[Hirokazu Takahashi]

Hi Chandra,

Why don't you implement CKRM memory controller as virtual NUMA
node.

I think what you want do is almost what NUMA code does, which
restricts resources to use. If you define virtual NUMA node with
some memory and some virtual CPUs, you can just assign target jobs
to them.

What do you think of my idea?

Thanks,
Hirokazu Takahashi.

> I am looking for improvement suggestions
>         - to not have a field in the page data structure for the mem
>           controller
> 	- to make vmscan.c cleaner.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Virtual NUMA machine and CKRM

[Martin J. Bligh]

> Why don't you implement CKRM memory controller as virtual NUMA
> node.
> 
> I think what you want do is almost what NUMA code does, which
> restricts resources to use. If you define virtual NUMA node with
> some memory and some virtual CPUs, you can just assign target jobs
> to them.
> 
> What do you think of my idea?

Please don't do that. For one, it'll make a mess of the NUMA code.
For another, it'll get really complicated once you have CKRM on top
of a NUMA system. Don't confuse the physical aspects of the machines
(its NUMA topology) with virtual resource limits.

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Virtual NUMA machine and CKRM

[Hirokazu Takahashi]

Hi,

> > Why don't you implement CKRM memory controller as virtual NUMA
> > node.
> > 
> > I think what you want do is almost what NUMA code does, which
> > restricts resources to use. If you define virtual NUMA node with
> > some memory and some virtual CPUs, you can just assign target jobs
> > to them.
> > 
> > What do you think of my idea?
> 
> Please don't do that. For one, it'll make a mess of the NUMA code.
> For another, it'll get really complicated once you have CKRM on top
> of a NUMA system. Don't confuse the physical aspects of the machines
> (its NUMA topology) with virtual resource limits.

I guessed you might say that.

But I feel the kernel would become much messier than that if CKRM is
implemented independently of the NUMA code, having the similar code
redundantly.

Though I'm not sure if it's the best way, it would be worth investigating.

> M.

Thanks,
Hirokazu Takahashi.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [ckrm-tech] Virtual NUMA machine and CKRM

[Dave Hansen]

On Fri, 2005-05-27 at 22:16 +0900, Hirokazu Takahashi wrote:
> Why don't you implement CKRM memory controller as virtual NUMA
> node.
> 
> I think what you want do is almost what NUMA code does, which
> restricts resources to use. If you define virtual NUMA node with
> some memory and some virtual CPUs, you can just assign target jobs
> to them.
> 
> What do you think of my idea?

First of all, NUMA nodes don't have any balancing done on them, so I
don't think they're an appropriate structure.  But, NUMA nodes *do*
contain zones, which are a slightly more appropriate structure.

One thing I pointed out when he first posted this code was that a lot of
the accounting gets shifted from the 'struct zone' to the ckrm class.
It was appropriate to have a set of macros to set up and perform this
indirection.

However, a 'struct zone' currently has more than one job.  It collects
"like" pages together, it provides accounting for those pages, and it
represents a contiguous area of memory.

If you could collect just the accounting pieces out of 'struct zone',
perhaps those could be used by both ckrm classes, and the old 'struct
zone'.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [ckrm-tech] Virtual NUMA machine and CKRM

[Chandra Seetharaman]

On Fri, May 27, 2005 at 07:15:58AM -0700, Dave Hansen wrote:
> On Fri, 2005-05-27 at 22:16 +0900, Hirokazu Takahashi wrote:
> > Why don't you implement CKRM memory controller as virtual NUMA
> > node.
> > 
> > I think what you want do is almost what NUMA code does, which
> > restricts resources to use. If you define virtual NUMA node with
> > some memory and some virtual CPUs, you can just assign target jobs
> > to them.
> > 
> > What do you think of my idea?
> 
> First of all, NUMA nodes don't have any balancing done on them, so I
> don't think they're an appropriate structure.  But, NUMA nodes *do*
> contain zones, which are a slightly more appropriate structure.
> 
> One thing I pointed out when he first posted this code was that a lot of
> the accounting gets shifted from the 'struct zone' to the ckrm class.
> It was appropriate to have a set of macros to set up and perform this
> indirection.
> 
> However, a 'struct zone' currently has more than one job.  It collects
> "like" pages together, it provides accounting for those pages, and it
> represents a contiguous area of memory.
> 
> If you could collect just the accounting pieces out of 'struct zone',
> perhaps those could be used by both ckrm classes, and the old 'struct
> zone'.

will look into it.

Thanks
> 
> -- Dave
> 

-- 

----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - sekharan@us.ibm.com   |      .......you may get it.
----------------------------------------------------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Virtual NUMA machine and CKRM

[Chandra Seetharaman]

On Fri, May 27, 2005 at 10:16:13PM +0900, Hirokazu Takahashi wrote:
> Hi Chandra,
> 
> Why don't you implement CKRM memory controller as virtual NUMA
> node.
> 
> I think what you want do is almost what NUMA code does, which
> restricts resources to use. If you define virtual NUMA node with

Besides mixing virtual/physical, IMHO it will be hairy as the ckrm class
needs to have all the zones that exists in the system(unlike the real NUMA
node), and it will get more complicated when CKRM is deployed in a NUMA
system itself.


> some memory and some virtual CPUs, you can just assign target jobs
> to them.
> 
> What do you think of my idea?
> 
> Thanks,
> Hirokazu Takahashi.
> 
> > I am looking for improvement suggestions
> >         - to not have a field in the page data structure for the mem
> >           controller
> > 	- to make vmscan.c cleaner.

-- 

----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - sekharan@us.ibm.com   |      .......you may get it.
----------------------------------------------------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [ckrm-tech] Virtual NUMA machine and CKRM

[YAMAMOTO Takashi]

hi,

> Why don't you implement CKRM memory controller as virtual NUMA
> node.

i think that you need to describe what's a "virtual NUMA node" before
soliciting comments.

> I think what you want do is almost what NUMA code does, which
> restricts resources to use. If you define virtual NUMA node with
> some memory and some virtual CPUs, you can just assign target jobs
> to them.

because resource restrictions from numa and ckrm are very different,
i don't think there is a single effective structure for both of them.

YAMAMOTO Takashi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>