use of alloc_bootmem for a PCI-e device

use of alloc_bootmem for a PCI-e device

[Jean-Francois Dagenais]

Hello fellow hackers,

I am maintaining a kernel for an embedded product. We have an FPGA
acquisition device interfaced through PCI-e on different intel platforms.
The acquisition device needs an extra large physically contiguous memory
area to autonomously dump acquired data into.

In a previous product, VT-d was available and I made use of it to allocate
128MB+ using vmalloc, then mapping it so it forms a contiguous address
chunk from the device side.

We are doing another incarnation of the product on an Atom E6xx which has
no such IOMMU and am looking into ways of allocating a huge chunk of ram.
Kind of like integrated gfx chips do with RAM, but I don't have the assistance
of the BIOS.

Based on suggestions to try using alloc_bootmem, I have started looking into
it by first making a platform (under arch/x86/platform) built-in module using
"pure_initcall" to get an early hook to allocate this memory. This approach
failed because the call would happen after SLUB init.

I then proceeded to hack "mm_init()" in init/main.c so that I do the alloc_bootmem
call after "mem_init()" but before "kmem_cache_init()". I successfully get the huge
chunk I request, and test that it is really physically contiguous.

My joy was not long lived, it was too easy. Once fully booted, I load a module
which tests a patter I wrote in memory to see if the memory got touched since
the early init moment, and it did. A crude test, but right away tells me the
allocation was not respected.

Any thoughts on how to achieve this?

We are targeting v3.1 64 or 32 bits, testing with 3.2-rc1 on x84_64. I noticed this
means "nobootmem.c" is used.

Thanks
/jfd
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: use of alloc_bootmem for a PCI-e device

[Michal Nazarewicz]

On Wed, 23 Nov 2011 20:30:30 +0100, Jean-Francois Dagenais <jeff.dagenais@gmail.com> wrote:
> I am maintaining a kernel for an embedded product. We have an FPGA
> acquisition device interfaced through PCI-e on different intel platforms.
> The acquisition device needs an extra large physically contiguous memory
> area to autonomously dump acquired data into.

[...]

> We are doing another incarnation of the product on an Atom E6xx which has
> no such IOMMU and am looking into ways of allocating a huge chunk of ram.
> Kind of like integrated gfx chips do with RAM, but I don't have the assistance
> of the BIOS.

One trick that you might try to use (even though it's a bit hackish) is to
pass ram=### on Linux command line where the number passed is actual memory
minus size of the buffer you need.  Other then that, you might take a look
at CMA (CMAv17 it was sent last week or so to linux-mm) which in one of the
initialisation steps needs to grab memory regions.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +----<email/xmpp: mpn@google.com>--------------ooO--(_)--Ooo--

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: use of alloc_bootmem for a PCI-e device

[Jean-Francois Dagenais]

On Nov 23, 2011, at 14:31, Michal Nazarewicz wrote:

> On Wed, 23 Nov 2011 20:30:30 +0100, Jean-Francois Dagenais <jeff.dagenais@gmail.com> wrote:
>> I am maintaining a kernel for an embedded product. We have an FPGA
>> acquisition device interfaced through PCI-e on different intel platforms.
>> The acquisition device needs an extra large physically contiguous memory
>> area to autonomously dump acquired data into.
> 
> [...]
> 
>> We are doing another incarnation of the product on an Atom E6xx which has
>> no such IOMMU and am looking into ways of allocating a huge chunk of ram.
>> Kind of like integrated gfx chips do with RAM, but I don't have the assistance
>> of the BIOS.
> 
> One trick that you might try to use (even though it's a bit hackish) is to
> pass ram=### on Linux command line where the number passed is actual memory
> minus size of the buffer you need.  Other then that, you might take a look
> at CMA (CMAv17 it was sent last week or so to linux-mm) which in one of the
> initialisation steps needs to grab memory regions.
Thanks for that, I am looking into it.

Looks like it can do what I want. Are there any mainline merge plans?

Unless I am mistaken, because of SWIOTLB, only x86_32 is supported, correct?

Since I want to allocate the buffer once at startup, then keep it until shutdown,
can you suggest a simpler, less flexible alternative? (other than the boot args
method which I consider a fallback plan).

> 
> -- 
> Best regards,                                         _     _
> .o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
> ..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
> ooo +----<email/xmpp: mpn@google.com>--------------ooO--(_)--Ooo--
Thanks for helping!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: use of alloc_bootmem for a PCI-e device

[Michal Nazarewicz]

> On Nov 23, 2011, at 14:31, Michal Nazarewicz wrote:
>> One trick that you might try to use (even though it's a bit hackish) is to
>> pass ram=### on Linux command line where the number passed is actual memory
>> minus size of the buffer you need.  Other then that, you might take a look
>> at CMA (CMAv17 it was sent last week or so to linux-mm) which in one of the
>> initialisation steps needs to grab memory regions.

On Wed, 23 Nov 2011 22:21:15 +0100, Jean-Francois Dagenais <jeff.dagenais@gmail.com> wrote:
> Looks like it can do what I want. Are there any mainline merge plans?

There are plans...  Execution is uncertain. ;)

> Unless I am mistaken, because of SWIOTLB, only x86_32 is supported, correct?
>
> Since I want to allocate the buffer once at startup, then keep it until shutdown,
> can you suggest a simpler, less flexible alternative?

Oh no, I wouldn't recommend using the full CMA for your purpose, but no matter
there is a piece of code that does what you need.  Marek has added support for
Intel so it should work for you as well, even though I have had a chance to
run that piece yet.

You are interested in parts of patch 8[1] (namely dma_declare_contiguous()
function) and the last hunk of patch 9[2] (namely the part where
dma_contiguous_reserve() is called).

[1] http://article.gmane.org/gmane.linux.kernel.mm/70321
[2] http://article.gmane.org/gmane.linux.kernel.mm/70318

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +----<email/xmpp: mpn@google.com>--------------ooO--(_)--Ooo--

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>