sparcemem or discontig?

sparcemem or discontig?

[C Michael Sundius]

Hi,

I'm working on a 32bit mips 1 cpu platform that has 2 large banks of
memory that are discontiguous in the physical address space. That is to
say there is a large hole in between them. We are using the 2.6.24
kernel. In order to save memory on page tables, we'd like to employ the
use of either the CONFIG_SPARCEMEM, or CONFIG_DISCONTIG configuration
options, but I'm a bit unsure which I should use.

My understanding is that SPARCEMEM is the way of the future, and since
I don't really have a NUMA machine, maybe sparcemem is more appropriate,
yes? On the other hand I can't find much info about how it works or how
to add support for it on an architecture that has here-to-fore not
supported that option.

Is there anywhere that there is a paper or rfp that describes how the
spacemem (or discontig) features work (and/or the differences between
then)? Has anyone out there done this for MIPS32? has anyone had
experience with adding support for either sparcemem or discontig on an
arch before , that could give me a their thoughts the process or "gotchas"?

thanks
Mike



     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparcemem or discontig?

[Dave Hansen]

On Tue, 2008-07-29 at 11:11 -0700, C Michael Sundius wrote:
> 
> My understanding is that SPARCEMEM is the way of the future, and since
> I don't really have a NUMA machine, maybe sparcemem is more appropriate,
> yes? On the other hand I can't find much info about how it works or how
> to add support for it on an architecture that has here-to-fore not
> supported that option.
> 
> Is there anywhere that there is a paper or rfp that describes how the
> spacemem (or discontig) features work (and/or the differences between
> then)?

I think you're talking about sparsemem. :)

My opinion is that NUMA and DISCONTIG are too intertwined to be useful
apart from the other.  I use sparsemem on my non-NUMA 2 CPU laptop since
it has a 1GB hole.  It is *possible* to use DISCONTIG without NUMA, and
I'm sure people use it this way, but I just personally think it is a bit
of a pain.  

Basically, to add sparsemem support for an architecture, you need a
header like these:

dave@nimitz:~/lse/linux/2.5/linux-2.6.git$ find | grep sparse | xargs
grep -c '^.*$'
./include/asm-powerpc/sparsemem.h:32
./include/asm-x86/sparsemem.h:34
./include/asm-sh/sparsemem.h:16
./include/asm-mips/sparsemem.h:14
./include/asm-ia64/sparsemem.h:20
./include/asm-s390/sparsemem.h:18
./include/asm-arm/sparsemem.h:10

These are generally pretty darn small (the largest is 34 lines).  You
also need to tweak some things in your per-arch Kconfig.  ARM looks like
a pretty simple use of sparsemem.  You might want to start with what
they've done.  We tried really, really hard to make it easy to add to
new architectures.

Feel free to cc me and Andy (cc'd) on the patches that you come up with.
I'd be happy to sanity check them for you.  If *you* want to document
the process for the next guy, I'm sure we'd be able to find some spot in
Documentation/ so the next guy has an easier time. :)

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparcemem or discontig?

[Andy Whitcroft]

On Tue, Jul 29, 2008 at 02:51:21PM -0700, Dave Hansen wrote:
> On Tue, 2008-07-29 at 11:11 -0700, C Michael Sundius wrote:
> > 
> > My understanding is that SPARCEMEM is the way of the future, and since
> > I don't really have a NUMA machine, maybe sparcemem is more appropriate,
> > yes? On the other hand I can't find much info about how it works or how
> > to add support for it on an architecture that has here-to-fore not
> > supported that option.
> > 
> > Is there anywhere that there is a paper or rfp that describes how the
> > spacemem (or discontig) features work (and/or the differences between
> > then)?
> 
> I think you're talking about sparsemem. :)
> 
> My opinion is that NUMA and DISCONTIG are too intertwined to be useful
> apart from the other.  I use sparsemem on my non-NUMA 2 CPU laptop since
> it has a 1GB hole.  It is *possible* to use DISCONTIG without NUMA, and
> I'm sure people use it this way, but I just personally think it is a bit
> of a pain.  
> 
> Basically, to add sparsemem support for an architecture, you need a
> header like these:
> 
> dave@nimitz:~/lse/linux/2.5/linux-2.6.git$ find | grep sparse | xargs
> grep -c '^.*$'
> ./include/asm-powerpc/sparsemem.h:32
> ./include/asm-x86/sparsemem.h:34
> ./include/asm-sh/sparsemem.h:16
> ./include/asm-mips/sparsemem.h:14
> ./include/asm-ia64/sparsemem.h:20
> ./include/asm-s390/sparsemem.h:18
> ./include/asm-arm/sparsemem.h:10
> 
> These are generally pretty darn small (the largest is 34 lines).  You
> also need to tweak some things in your per-arch Kconfig.  ARM looks like
> a pretty simple use of sparsemem.  You might want to start with what
> they've done.  We tried really, really hard to make it easy to add to
> new architectures.
> 
> Feel free to cc me and Andy (cc'd) on the patches that you come up with.
> I'd be happy to sanity check them for you.  If *you* want to document
> the process for the next guy, I'm sure we'd be able to find some spot in
> Documentation/ so the next guy has an easier time. :)

Always interested in new users of sparsemem.  Cc me :).

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparcemem or discontig?

[C Michael Sundius]

Andy Whitcroft wrote:
> On Tue, Jul 29, 2008 at 02:51:21PM -0700, Dave Hansen wrote:
>   
>> On Tue, 2008-07-29 at 11:11 -0700, C Michael Sundius wrote:
>>     
>>> My understanding is that SPARCEMEM is the way of the future, and since
>>> I don't really have a NUMA machine, maybe sparcemem is more appropriate,
>>> yes? On the other hand I can't find much info about how it works or how
>>> to add support for it on an architecture that has here-to-fore not
>>> supported that option.
>>>
>>> Is there anywhere that there is a paper or rfp that describes how the
>>> spacemem (or discontig) features work (and/or the differences between
>>> then)?
>>>       
>> I think you're talking about sparsemem. :)
>>
>> My opinion is that NUMA and DISCONTIG are too intertwined to be useful
>> apart from the other.  I use sparsemem on my non-NUMA 2 CPU laptop since
>> it has a 1GB hole.  It is *possible* to use DISCONTIG without NUMA, and
>> I'm sure people use it this way, but I just personally think it is a bit
>> of a pain.  
>>
>> Basically, to add sparsemem support for an architecture, you need a
>> header like these:
>>
>> dave@nimitz:~/lse/linux/2.5/linux-2.6.git$ find | grep sparse | xargs
>> grep -c '^.*$'
>> ./include/asm-powerpc/sparsemem.h:32
>> ./include/asm-x86/sparsemem.h:34
>> ./include/asm-sh/sparsemem.h:16
>> ./include/asm-mips/sparsemem.h:14
>> ./include/asm-ia64/sparsemem.h:20
>> ./include/asm-s390/sparsemem.h:18
>> ./include/asm-arm/sparsemem.h:10
>>
>> These are generally pretty darn small (the largest is 34 lines).  You
>> also need to tweak some things in your per-arch Kconfig.  ARM looks like
>> a pretty simple use of sparsemem.  You might want to start with what
>> they've done.  We tried really, really hard to make it easy to add to
>> new architectures.
>>     
Pardon my ignorance, but is sparcemem independent of the bootmem allocator?

We also use highmem. I noticed that all of our kmap and kmap_atomic code 
is located
in the arch/mips directory. Is the sparcemem also independent of that? 
should I expect
that I will have to make some changes in that...
>> Feel free to cc me and Andy (cc'd) on the patches that you come up with.
>> I'd be happy to sanity check them for you.  If *you* want to document
>> the process for the next guy, I'm sure we'd be able to find some spot in
>> Documentation/ so the next guy has an easier time. :)
>>     
I'm happy to write a how to for sparcemem.



     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparcemem or discontig?

[Dave Hansen]

On Wed, 2008-07-30 at 09:23 -0700, C Michael Sundius wrote:
> Pardon my ignorance, but is sparcemem independent of the bootmem allocator?

Yes, it really don't have much to do with bootmem.

> We also use highmem. I noticed that all of our kmap and kmap_atomic code 
> is located
> in the arch/mips directory. Is the sparcemem also independent of that? 
> should I expect
> that I will have to make some changes in that...

No, I don't think you'll need any changes to those.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Turning on Sparsemem

[C Michael Sundius]

Hi Andy and Dave,

I turned on sparsemem as you described before. I am crashing in
the mem_init() function when I try a call to pfn_to_page().

I've noticed that that macro uses the sparsemem macro 
__pfn_to_section(pfn) and
that intern calls __nr_to_section(nr). That finally looks at the 
mem_section[] variable.
well.. this returns NULL since it seems as though my mem_section[] array 
looks
to be not initialized correctly.

QUESTION: where does this array get initialized. I've looked through the 
code and
can't seem to see how that is initialized.

recall I'm using mips32 processor, but I've looked in all the processors.
it seems as though sparse_init() and memory present() both use 
__nr_to_section()
and thus would require mem_section[] to be set up already.

thanks for your help
Mike



     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Turning on Sparsemem

[Dave Hansen]

On Tue, 2008-08-05 at 14:39 -0700, C Michael Sundius wrote:
> Hi Andy and Dave,
> 
> I turned on sparsemem as you described before. I am crashing in
> the mem_init() function when I try a call to pfn_to_page().
> 
> I've noticed that that macro uses the sparsemem macro 
> __pfn_to_section(pfn) and
> that intern calls __nr_to_section(nr). That finally looks at the 
> mem_section[] variable.
> well.. this returns NULL since it seems as though my mem_section[] array 
> looks
> to be not initialized correctly.
> 
> QUESTION: where does this array get initialized. I've looked through the 
> code and
> can't seem to see how that is initialized.

My first guess is that you're missing a call to sparse_init() in your
architecture-specific code.  On x86_32, we do that in paging_init(),
just before zone_sizes_init() (arch/x86/mm/init_32.c).

Before you call this, you'll also have to call memory_present(...) on
the memory that you do have.  But, you should probably already have done
that.

> recall I'm using mips32 processor, but I've looked in all the processors.
> it seems as though sparse_init() and memory present() both use 
> __nr_to_section()
> and thus would require mem_section[] to be set up already.

__nr_to_section() is special.  It takes a section number and just gives
you back the 'struct mem_section'.  It doesn't actually look into that
mem_section and make sure it is valid, it just locates the data
structure.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Turning on Sparsemem

[C Michael Sundius]

Dave Hansen wrote:
> On Tue, 2008-08-05 at 14:39 -0700, C Michael Sundius wrote:
>   
>> Hi Andy and Dave,
>>
>> I turned on sparsemem as you described before. I am crashing in
>> the mem_init() function when I try a call to pfn_to_page().
>>
>> I've noticed that that macro uses the sparsemem macro 
>> __pfn_to_section(pfn) and
>> that intern calls __nr_to_section(nr). That finally looks at the 
>> mem_section[] variable.
>> well.. this returns NULL since it seems as though my mem_section[] array 
>> looks
>> to be not initialized correctly.
>>
>> QUESTION: where does this array get initialized. I've looked through the 
>> code and
>> can't seem to see how that is initialized.
>>     
>
> My first guess is that you're missing a call to sparse_init() in your
> architecture-specific code.  On x86_32, we do that in paging_init(),
> just before zone_sizes_init() (arch/x86/mm/init_32.c).
>
> Before you call this, you'll also have to call memory_present(...) on
> the memory that you do have.  But, you should probably already have done
> that.
>
>   
yes I call sparse_init()..... My code might be not calling 
memory_present for some of the memory
I'm looking at that right now.
>> recall I'm using mips32 processor, but I've looked in all the processors.
>> it seems as though sparse_init() and memory present() both use 
>> __nr_to_section()
>> and thus would require mem_section[] to be set up already.
>>     
>
> __nr_to_section() is special.  It takes a section number and just gives
> you back the 'struct mem_section'.  It doesn't actually look into that
> mem_section and make sure it is valid, it just locates the data
> structure.
>
> -- Dave
>
>   
still that code is strange to me:

-------------code
static inline struct mem_section *__nr_to_section(unsigned long nr)
{
        if (!mem_section[SECTION_NR_TO_ROOT(nr)])
                return NULL;
        return &mem_section[SECTION_NR_TO_ROOT(nr)][nr & SECTION_ROOT_MASK];
}

--------------

on the first line of the function above, what does it mean "if not 
<struct>"?  seems that returns true if
the contents of that struct is "0"... but either way, doesn't that have 
to be initialized to something before
it is called from memory_present()?

mike



     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Turning on Sparsemem

[Dave Hansen]

On Tue, 2008-08-05 at 15:04 -0700, C Michael Sundius wrote:
> still that code is strange to me:
> 
> -------------code
> static inline struct mem_section *__nr_to_section(unsigned long nr)
> {
>         if (!mem_section[SECTION_NR_TO_ROOT(nr)])
>                 return NULL;
>         return &mem_section[SECTION_NR_TO_ROOT(nr)][nr & SECTION_ROOT_MASK];
> }
> 
> --------------
> 
> on the first line of the function above, what does it mean "if not 
> <struct>"?  seems that returns true if
> the contents of that struct is "0"... but either way, doesn't that have 
> to be initialized to something before
> it is called from memory_present()?

Yeah, this is confusing code.  The goal here was to not have any #ifdefs
for the normal vs. extreme cases.  In the !EXTREME case, the
mem_section[] array is statically allocated and that first check gets
optimized out completely.  We check for it, but the compiler kills that
check.  

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>