sparsemem & sparsemem extreme question

sparsemem & sparsemem extreme question

[Heiko Carstens]

Hi all,

I did an implementation of CONFIG_SPARSEMEM for s390, which indeed was quite
easy. Just to find out that it was not sufficient :)
SPARSEMEM_EXTREME looks better but unfortunately adds another layer of
indirection.
I'm just wondering why there is all this indirection stuff here and why not
have one contiguous aray of struct pages (residing in the vmalloc area) that
deals with whatever size of memory an architecture wants to support.
Unused areas just wouldn't have any backing with real pages and on access
generate a page fault (nobody is supposed to access these pages anyway).
This would have the advantage that all the primitives like e.g. pfn_to_page
would be as simple as before, no need to waste large parts of the page flags
and in addition it would easily allow for memory hotplug on page size
granularity.
The only drawbacks are (as far as I can see) a _huge_ virtual mem_map array,
but that shouldn't matter too much. A real problem could be that the mem_map
array and therefore the vmalloc area need to be generated quiete early.

Most probably this has already been thought about before, but I couldn't find
anything in the achives.

Heiko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Andy Whitcroft]

Heiko Carstens wrote:

> I did an implementation of CONFIG_SPARSEMEM for s390, which indeed was quite
> easy. Just to find out that it was not sufficient :)
> SPARSEMEM_EXTREME looks better but unfortunately adds another layer of
> indirection.
> I'm just wondering why there is all this indirection stuff here and why not
> have one contiguous aray of struct pages (residing in the vmalloc area) that
> deals with whatever size of memory an architecture wants to support.
> Unused areas just wouldn't have any backing with real pages and on access
> generate a page fault (nobody is supposed to access these pages anyway).
> This would have the advantage that all the primitives like e.g. pfn_to_page
> would be as simple as before, no need to waste large parts of the page flags
> and in addition it would easily allow for memory hotplug on page size
> granularity.
> The only drawbacks are (as far as I can see) a _huge_ virtual mem_map array,
> but that shouldn't matter too much. A real problem could be that the mem_map
> array and therefore the vmalloc area need to be generated quiete early.
> 
> Most probably this has already been thought about before, but I couldn't find
> anything in the achives.

During the implementation of SPARSEMEM_EXTREME other layouts such as the
huge 'partially populated' mem_map were considered.  For a number of our
target architectures kernel virtual address is at a premium so this
would not be suitable for them.  We did consider whether to have
different mechanisms for KVA rich architectures but (if I remember
correctly) benchmarking the implementation seemed to indicate that the
additional indirection was insignificant if even detectable.

The architecture of sparsemem is supposed to allow architecture specific
implementations should that be necessary but I've not yet seen a
compelling arguement for one yet.

On the subject of page flags, I would point out that SPARSEMEM either
reuses already used bits for 32 bit architectures, or makes use of
unused bits in the 64 case.  It doesn't reduce the number of flags bits
available.

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Dave Hansen]

On Tue, 2005-10-04 at 08:50 +0200, Heiko Carstens wrote:
> I'm just wondering why there is all this indirection stuff here and why not
> have one contiguous aray of struct pages (residing in the vmalloc area) that
> deals with whatever size of memory an architecture wants to support.

This is exactly what ia64 does today.  Programatically, it does remove a
layer of indirection.  However, there are some data structures that have
to be traversed during a lookup: the page tables.  Granted, the TLB will
provide some caching, but a lookup on ia64 can potentially be much more
expensive than the two cacheline misses that sparsemem extreme might
have.

In the end no one has ever produced any compelling performance reason to
use a vmem_map (as ia64 calls it).  In addition, sparsemem doesn't cause
any known performance regressions, either.  

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Heiko Carstens]

> > I'm just wondering why there is all this indirection stuff here and why not
> > have one contiguous aray of struct pages (residing in the vmalloc area) that
> > deals with whatever size of memory an architecture wants to support.
> This is exactly what ia64 does today.  Programatically, it does remove a
> layer of indirection.  However, there are some data structures that have
> to be traversed during a lookup: the page tables.  Granted, the TLB will
> provide some caching, but a lookup on ia64 can potentially be much more
> expensive than the two cacheline misses that sparsemem extreme might
> have.

Sure, just that on s390 we have a 1:1 mapping anyway. So these lookups would
be more or less for free for us (compared to what we have now).

> In the end no one has ever produced any compelling performance reason to
> use a vmem_map (as ia64 calls it).  In addition, sparsemem doesn't cause
> any known performance regressions, either.

As far as I understand the memory hotplug patches they won't work without
SPARSEMEM support. So the ia64 approach with a vmem_map will not work here,
right?

Actually my concern is that whenever the address space that is covered with
SPARSEMEM_EXTREME is not sufficient just another layer of indirection needs
to be added.

Heiko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Dave Hansen]

On Wed, 2005-10-05 at 08:39 +0200, Heiko Carstens wrote:
> > > I'm just wondering why there is all this indirection stuff here and why not
> > > have one contiguous aray of struct pages (residing in the vmalloc area) that
> > > deals with whatever size of memory an architecture wants to support.
> > This is exactly what ia64 does today.  Programatically, it does remove a
> > layer of indirection.  However, there are some data structures that have
> > to be traversed during a lookup: the page tables.  Granted, the TLB will
> > provide some caching, but a lookup on ia64 can potentially be much more
> > expensive than the two cacheline misses that sparsemem extreme might
> > have.
> 
> Sure, just that on s390 we have a 1:1 mapping anyway. So these lookups would
> be more or less for free for us (compared to what we have now).

Is the 1:1 mapping done with pagetables?  If so, it is not free.

> > In the end no one has ever produced any compelling performance reason to
> > use a vmem_map (as ia64 calls it).  In addition, sparsemem doesn't cause
> > any known performance regressions, either.
> 
> As far as I understand the memory hotplug patches they won't work without
> SPARSEMEM support. So the ia64 approach with a vmem_map will not work here,
> right?

If we had vmem_map implemented for every arch that supported memory
hotplug, and it didn't have performance implications, then we could have
vmem_map everywhere, and use it for hotplug.

> Actually my concern is that whenever the address space that is covered with
> SPARSEMEM_EXTREME is not sufficient just another layer of indirection needs
> to be added.

Do you have any performance numbers to back up your concerns, or is it
more about the code complexity?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Heiko Carstens]

> > > to be traversed during a lookup: the page tables.  Granted, the TLB will
> > > provide some caching, but a lookup on ia64 can potentially be much more
> > > expensive than the two cacheline misses that sparsemem extreme might
> > > have.
> > Sure, just that on s390 we have a 1:1 mapping anyway. So these lookups would
> > be more or less for free for us (compared to what we have now).
> Is the 1:1 mapping done with pagetables?  If so, it is not free.

Sure, it's done with pagetables. What I meant: we have the 1:1 mapping already
today. So adding anything to the vmalloc area won't make it more expensive.

> > Actually my concern is that whenever the address space that is covered with
> > SPARSEMEM_EXTREME is not sufficient just another layer of indirection needs
> > to be added.
> Do you have any performance numbers to back up your concerns, or is it
> more about the code complexity?

No, my concern is actually that the s390 archticture actually will come up
with some sort of memory that's present in the physical address space where
the most significant bit of the addresses will be turned _on_. That means we
would need to support the whole 64 bit physical address space...
Considering this, this would be good for at least one if not two additional
indirection layers, which would make the code too complex, IMHO.
That's why I think the vmem_map approach would be easiest to implement this :)

Heiko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Dave Hansen]

On Wed, 2005-10-05 at 17:58 +0200, Heiko Carstens wrote:
> we have the 1:1 mapping already
> today. So adding anything to the vmalloc area won't make it more expensive.

The pagetables themselves have a cost, as do the lookups.  Those make it
more expensive.  On 64-bit machines the vaddr space is not an expense.

> No, my concern is actually that the s390 archticture actually will come up
> with some sort of memory that's present in the physical address space where
> the most significant bit of the addresses will be turned _on_.

Why do you think this?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Heiko Carstens]

> > No, my concern is actually that the s390 archticture actually will come up
> > with some sort of memory that's present in the physical address space where
> > the most significant bit of the addresses will be turned _on_.
> 
> Why do you think this?

That's a matter of fact and what the Specs say...

Heiko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Dave Hansen]

On Wed, 2005-10-05 at 18:10 +0200, Heiko Carstens wrote:
> > > No, my concern is actually that the s390 archticture actually will come up
> > > with some sort of memory that's present in the physical address space where
> > > the most significant bit of the addresses will be turned _on_.
> > 
> > Why do you think this?
> 
> That's a matter of fact and what the Specs say...

I'd appreciate any pointer to the relevant information, especially the
stuff that explains just how sparse a physical address space can be on
that architecture.  What would discontigmem have done with the same
layout?  Does s390 even support discontigmem?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Heiko Carstens]

> > That's a matter of fact and what the Specs say...
> I'd appreciate any pointer to the relevant information, especially the
> stuff that explains just how sparse a physical address space can be on
> that architecture.  What would discontigmem have done with the same
> layout?  Does s390 even support discontigmem?

s390 does not support discontigmem at all. And unfortunately the
documentation is not publicly available yet, sorry.
Anything specific you need to know about the memory layout?

Heiko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Dave Hansen]

On Wed, 2005-10-05 at 19:12 +0200, Heiko Carstens wrote:
> > > That's a matter of fact and what the Specs say...
> > I'd appreciate any pointer to the relevant information, especially the
> > stuff that explains just how sparse a physical address space can be on
> > that architecture.  What would discontigmem have done with the same
> > layout?  Does s390 even support discontigmem?
> 
> s390 does not support discontigmem at all. And unfortunately the
> documentation is not publicly available yet, sorry.
> Anything specific you need to know about the memory layout?

How sparse is it?  How few present pages can be there be in a worst-case
physical area?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Heiko Carstens]

> > Anything specific you need to know about the memory layout?
> How sparse is it?  How few present pages can be there be in a worst-case
> physical area?

Worst case that is already currently valid is that you can have 1 MB
segments whereever you want in address space.

For instance I just configured a virtual machine that has the following
memory layout:

Address Range
-----------------------------------
0000000000000000 - 00000000000FFFFF
000000FFC0000000 - 000000FFC00FFFFF

Even though it's currently not possible to define memory segments above
1TB, this limit is likely to go away.
In addition if running in a logical partition we always have a small gap
at the 2GB barrier. Not sure how large that gap is, I'll check tomorrow.

Heiko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Dave Hansen]

On Wed, 2005-10-05 at 19:45 +0200, Heiko Carstens wrote:
> > > Anything specific you need to know about the memory layout?
> > How sparse is it?  How few present pages can be there be in a worst-case
> > physical area?
> 
> Worst case that is already currently valid is that you can have 1 MB
> segments whereever you want in address space.
...
> Even though it's currently not possible to define memory segments above
> 1TB, this limit is likely to go away.

Go away, or get moved up?

ia64 today is designed to work with 50 bits of physical address space,
and 30 bit sections.  That's exactly the same scale that you're talking
about with 1MB sections and 1TB of physical space.  So, sparsemem
extreme should be perfectly fine for that case (that's explicitly what
it was designed for).

How much bigger than 1TB will it go?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Heiko Carstens]

> > > > Anything specific you need to know about the memory layout?
> > > How sparse is it?  How few present pages can be there be in a worst-case
> > > physical area?
> > 
> > Worst case that is already currently valid is that you can have 1 MB
> > segments whereever you want in address space.
> ...
> > Even though it's currently not possible to define memory segments above
> > 1TB, this limit is likely to go away.
> 
> Go away, or get moved up?
> 
> ia64 today is designed to work with 50 bits of physical address space,
> and 30 bit sections.  That's exactly the same scale that you're talking
> about with 1MB sections and 1TB of physical space.  So, sparsemem
> extreme should be perfectly fine for that case (that's explicitly what
> it was designed for).
> 
> How much bigger than 1TB will it go?

As already mentioned, we will have physical memory with the MSB set. Afaik
the hardware uses this bit to distinguish between different types of memory.
So we are going to have the full 64 bit address space.

Heiko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Bob Picco]

Heiko Carstens wrote:	[Wed Oct 05 2005, 02:04:43PM EDT]
> > > > > Anything specific you need to know about the memory layout?
> > > > How sparse is it?  How few present pages can be there be in a worst-case
> > > > physical area?
> > > 
> > > Worst case that is already currently valid is that you can have 1 MB
> > > segments whereever you want in address space.
> > ...
> > > Even though it's currently not possible to define memory segments above
> > > 1TB, this limit is likely to go away.
> > 
> > Go away, or get moved up?
> > 
> > ia64 today is designed to work with 50 bits of physical address space,
> > and 30 bit sections.  That's exactly the same scale that you're talking
> > about with 1MB sections and 1TB of physical space.  So, sparsemem
> > extreme should be perfectly fine for that case (that's explicitly what
> > it was designed for).
> > 
> > How much bigger than 1TB will it go?
> 
> As already mentioned, we will have physical memory with the MSB set. Afaik
> the hardware uses this bit to distinguish between different types of memory.
> So we are going to have the full 64 bit address space.
> 
> Heiko
Possibly the architects did a similar thing to ia64.  When bit 63 is set the
access in uncached. This doesn't seem relevant to sparsemem.  The attribute
could be stored in page structure flags or some other way.

bob

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Dave Hansen]

On Wed, 2005-10-05 at 20:04 +0200, Heiko Carstens wrote: 
> As already mentioned, we will have physical memory with the MSB set. Afaik
> the hardware uses this bit to distinguish between different types of memory.
> So we are going to have the full 64 bit address space.

Is it just the MSB?  If so, we can probably just shift it down to some
reasonable address.  The only issue comes if you really have the whole
address space used in some random way.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem & sparsemem extreme question

[Heiko Carstens]

> > As already mentioned, we will have physical memory with the MSB set. Afaik
> > the hardware uses this bit to distinguish between different types of memory.
> > So we are going to have the full 64 bit address space.
> Is it just the MSB?  If so, we can probably just shift it down to some
> reasonable address.  The only issue comes if you really have the whole
> address space used in some random way.

Unfortunately there is more than just the MSB. If the MSB is set then there
will be a 'model dependent' number of bits just below the MSB used to encode
whatever the hardware thinks is necessary. Also you cannot tell how these
bits will be set from an operating system's view. Best thing to do is to
assume nothing and leave the addresses alone.

Heiko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>