Use of __pa() with CONFIG_NONLINEAR

Use of __pa() with CONFIG_NONLINEAR

[Dave Hansen]

So, for CONFIG_NONLINEAR, we introduce a new indirection layer for
virtual to physical conversions (and the inverse as well).  Our
implementation uses some data structures to do this (patch is here:
http://lwn.net/Articles/79124/), and the side-effect is that we can't
use __pa() or __va() until after the initialization has run, which is
early in setup_arch().

But, there are quite a few things that obviously need physical addresses
earlier than that, such as cr3 initialization at compile-time.  So, in
Dave McCracken's patch, he introduced a new function: __boot_pa() that
does what the old __pa() did.

This is the largest and hardest to maintain part of the CONFIG_NONLINEAR
patch at this point, and I'd love to start merging bits of it back in. 
Would anybody object to a patch that just does this for a bunch of
architectures?

--- include/asm-i386/page.h.orig	2004-07-27 14:31:09.000000000 -0700
+++ include/asm-i386/page.h	2004-07-27 14:31:36.000000000 -0700
@@ -128,8 +128,10 @@ static __inline__ int get_order(unsigned
 #define PAGE_OFFSET		((unsigned long)__PAGE_OFFSET)
 #define VMALLOC_RESERVE		((unsigned long)__VMALLOC_RESERVE)
 #define MAXMEM			(-__PAGE_OFFSET-__VMALLOC_RESERVE)
-#define __pa(x)			((unsigned long)(x)-PAGE_OFFSET)
-#define __va(x)			((void *)((unsigned long)(x)+PAGE_OFFSET))
+#define __boot_pa(x)		((unsigned long)(x)-PAGE_OFFSET)
+#define __boot_va(x)		((void *)((unsigned long)(x)+PAGE_OFFSET))
+#define __pa(x)			__boot_pa(x)
+#define __va(x)			__boot_va(x)
 #define pfn_to_kaddr(pfn)      __va((pfn) << PAGE_SHIFT)
 #ifndef CONFIG_DISCONTIGMEM
 #define pfn_to_page(pfn)	(mem_map + (pfn))

We will, of course be overriding __{v,p}a() for any architectures that
we do nonlinear with, later on.

The only other thing I can think of is to make __pa() effectively the
boot-time, simple one, and make virt_to_phys() the more sophisticated
one that we override with nonlinear.  But, some architectures '#define
__pa() virt_to_phys()', while others do the opposite, so that approach
could be significantly more work.  

Does anybody really remember what the different between the __{p,v}a()
functions and the virt_to_phys() ones is supposed to be?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Use of __pa() with CONFIG_NONLINEAR

[Joel Schopp]

> This is the largest and hardest to maintain part of the CONFIG_NONLINEAR
> patch at this point, and I'd love to start merging bits of it back in. 
> Would anybody object to a patch that just does this for a bunch of
> architectures?

I like the idea but would suggest a comment with the #defines to better 
explain why there are two names for the same thing, how they might in 
fact be different things in the future, and which one to use where.

> 
> --- include/asm-i386/page.h.orig	2004-07-27 14:31:09.000000000 -0700
> +++ include/asm-i386/page.h	2004-07-27 14:31:36.000000000 -0700
> @@ -128,8 +128,10 @@ static __inline__ int get_order(unsigned
>  #define PAGE_OFFSET		((unsigned long)__PAGE_OFFSET)
>  #define VMALLOC_RESERVE		((unsigned long)__VMALLOC_RESERVE)
>  #define MAXMEM			(-__PAGE_OFFSET-__VMALLOC_RESERVE)
> -#define __pa(x)			((unsigned long)(x)-PAGE_OFFSET)
> -#define __va(x)			((void *)((unsigned long)(x)+PAGE_OFFSET))
> +#define __boot_pa(x)		((unsigned long)(x)-PAGE_OFFSET)
> +#define __boot_va(x)		((void *)((unsigned long)(x)+PAGE_OFFSET))
> +#define __pa(x)			__boot_pa(x)
> +#define __va(x)			__boot_va(x)
>  #define pfn_to_kaddr(pfn)      __va((pfn) << PAGE_SHIFT)
>  #ifndef CONFIG_DISCONTIGMEM
>  #define pfn_to_page(pfn)	(mem_map + (pfn))

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Use of __pa() with CONFIG_NONLINEAR

[Mike Kravetz]

On Tue, Jul 27, 2004 at 03:00:30PM -0700, Dave Hansen wrote:
> So, for CONFIG_NONLINEAR, we introduce a new indirection layer for
> virtual to physical conversions (and the inverse as well).  Our
> implementation uses some data structures to do this (patch is here:
> http://lwn.net/Articles/79124/), and the side-effect is that we can't
> use __pa() or __va() until after the initialization has run, which is
> early in setup_arch().
> 
> But, there are quite a few things that obviously need physical addresses
> earlier than that, such as cr3 initialization at compile-time.  So, in
> Dave McCracken's patch, he introduced a new function: __boot_pa() that
> does what the old __pa() did.
> 

As Joel Stated in another note, I think documentation/comments is a must.
My 'guess' is that anyone writing early init code is going to have a
difficult time here.  As you stated the memsection initialization would
happen in setup_arch().  For arch independent code this makes it pretty
straight forward as everything run before the setup_arch() can use
the old/offset method of calculating physical addresses and everything
after can use the new method.  However, the arch specific initialization
code is more difficult as it depends on where the call paths are relative
to setup/alloc_memsections within setup_arch().

When I was trying to get nonlinear working on a specific architecture,
I made the routines go through 'pointer to functions' and had the memsections
initialization code modify the pointer after initialization was complete.
In this way, I got the nonlinear code code up and working without changing
all the early __pa/__va calls.  Obviously, this is a hack that should not
be used as it introduces another level of indirection to performance
critical code.  However, it was easy and saved me the effort of all that
code inspection. :)

-- 
Mike
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Use of __pa() with CONFIG_NONLINEAR

[Martin J. Bligh]

>> So, for CONFIG_NONLINEAR, we introduce a new indirection layer for
>> virtual to physical conversions (and the inverse as well).  Our
>> implementation uses some data structures to do this (patch is here:
>> http://lwn.net/Articles/79124/), and the side-effect is that we can't
>> use __pa() or __va() until after the initialization has run, which is
>> early in setup_arch().
>> 
>> But, there are quite a few things that obviously need physical addresses
>> earlier than that, such as cr3 initialization at compile-time.  So, in
>> Dave McCracken's patch, he introduced a new function: __boot_pa() that
>> does what the old __pa() did.
>> 
> 
> As Joel Stated in another note, I think documentation/comments is a must.
> My 'guess' is that anyone writing early init code is going to have a
> difficult time here.  As you stated the memsection initialization would
> happen in setup_arch().  For arch independent code this makes it pretty
> straight forward as everything run before the setup_arch() can use
> the old/offset method of calculating physical addresses and everything
> after can use the new method.  However, the arch specific initialization
> code is more difficult as it depends on where the call paths are relative
> to setup/alloc_memsections within setup_arch().
> 
> When I was trying to get nonlinear working on a specific architecture,
> I made the routines go through 'pointer to functions' and had the memsections
> initialization code modify the pointer after initialization was complete.
> In this way, I got the nonlinear code code up and working without changing
> all the early __pa/__va calls.  Obviously, this is a hack that should not
> be used as it introduces another level of indirection to performance
> critical code.  However, it was easy and saved me the effort of all that
> code inspection. :)

Can someone explain the necessity to create the new address space? We don't
need it with the current holes between nodes, and from my discssions with
Andy, I'm now unconvinced it's necessary.

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Use of __pa() with CONFIG_NONLINEAR

[Dave Hansen]

On Wed, 2004-07-28 at 12:47, Martin J. Bligh wrote:
> Can someone explain the necessity to create the new address space? We don't
> need it with the current holes between nodes, and from my discssions with
> Andy, I'm now unconvinced it's necessary.

Actually, the new address space is quite separated from what I'm
proposing here.  I'd prefer to discuss that part when we have an
implementation surrounding it.  I can explain it now if you'd like, but
it's going to be a bit harder with no code.  

The reason we need boot-time __{p,v}a() macros is really quite separate
from the new (logical) address space.  These new macros are just so we
can assume flat addressing during boot or compile-time, before any
nonlinear structures are set up.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Use of __pa() with CONFIG_NONLINEAR

[Martin J. Bligh]

> On Wed, 2004-07-28 at 12:47, Martin J. Bligh wrote:
>> Can someone explain the necessity to create the new address space? We don't
>> need it with the current holes between nodes, and from my discssions with
>> Andy, I'm now unconvinced it's necessary.
> 
> Actually, the new address space is quite separated from what I'm
> proposing here.  I'd prefer to discuss that part when we have an
> implementation surrounding it.  I can explain it now if you'd like, but
> it's going to be a bit harder with no code.  
> 
> The reason we need boot-time __{p,v}a() macros is really quite separate
> from the new (logical) address space.  These new macros are just so we
> can assume flat addressing during boot or compile-time, before any
> nonlinear structures are set up.

Ah, OK ... makes more sense - thanks.

However ... what happens to functions calling __pa that are called from 
boot time and run time code?

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Use of __pa() with CONFIG_NONLINEAR

[Dave Hansen]

On Wed, 2004-07-28 at 13:15, Martin J. Bligh wrote:
> However ... what happens to functions calling __pa that are called from 
> boot time and run time code?

I've actually only run into one of those so far that I know of, and that
was on ppc64 (i386 had none that I found).  In that one case, I used an
if(unlikely()) to optimize for the run-time one.  There might be more,
but I think they're rare enough to just code it with an if() in each
case.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>