Making PCI Memory Cachable

Making PCI Memory Cachable

[John Fusco]

I have numerous custom PCI devices that implement SRAM or DRAM on the 
PCI bus. I would like to explore making this memory cachable in the 
hopes that writes to memory can be done from user space and will be done 
in bursts rather than single cycles.

Is this crazy or what? I assumed that the VM_IO flag in remap_pfn_range 
controls this, but that doesn't appear to be the case.

A simple test from user space is the following:
    ptr = mmap(...)
    memset(ptr, 'a', size)


The driver gets the pointer with remap_pfn_range, but I always see 
64-bit writes to memory (this is x86_64 architecture). If this were 
cachable and the HW waqs cooperative, I would expect to see cache line 
bursts (e.g. 128 bytes).

I think one of two things is happening:
    a) The kernel is not making this memory cachable
    b) The memory is cachable, but the chipset is throttling the bursts

Can someone enlighten me?

Thanks,
John

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Making PCI Memory Cachable

[Andi Kleen]

On Tuesday 28 November 2006 15:02, John Fusco wrote:
> I have numerous custom PCI devices that implement SRAM or DRAM on the 
> PCI bus. I would like to explore making this memory cachable in the 
> hopes that writes to memory can be done from user space and will be done 
> in bursts rather than single cycles.

You want write combining, not cacheable. The only way to do 
this currently is to set a MTRR. See Documentation/mtrr.txt 
by default.

>     b) The memory is cachable, but the chipset is throttling the bursts

The normal cache coherency protocol doesn't work over PCI

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Making PCI Memory Cachable

[John Fusco]

----- Original Message ----
  

From: Andi Kleen <ak@suse.de>
  

To: John Fusco <fusco_john@yahoo.com>
  

Cc: linux-mm@kvack.org
  

Sent: Wednesday, December 6, 2006 4:16:27 PM
  

Subject: Re: Making PCI Memory Cachable
  


  

On Tuesday 28 November 2006 15:02, John Fusco wrote:
  

> I have numerous custom PCI devices that implement SRAM or DRAM on the 
  

> PCI bus. I would like to explore making this memory cachable in the 
  

> hopes that writes to memory can be done from user space and will be done 
  

> in bursts rather than single cycles.
  


  

You want write combining, not cacheable. The only way to do 
  

this currently is to set a MTRR. See Documentation/mtrr.txt 
  

by default.
  


  

>     b) The memory is cachable, but the chipset is throttling the bursts
  


  

The normal cache coherency protocol doesn't work over PCI
  


  

-Andi

Thanks for the reply.

I am aware of the MTRRs, but I was hoping for a more elegant solution. 

BTW, why won't cache coherency protocol work over PCI? It has commands to support this, such as "memory read line" and "memory write line". Is it that Linux does not allow memory outside of RAM to be cacheable?

John








 
____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Making PCI Memory Cachable

[Andi Kleen]

> BTW, why won't cache coherency protocol work over PCI? 

It's not supported by the hardware. There is a initiative from Intel
to support it in the future over PCI-Express, but that's some time
off and still most devices won't support it.

> It has commands to support this, such as "memory read line" and "memory write line". Is it that Linux does not allow memory outside of RAM to be cacheable? 

A proper cache coherency protocol is much more complicated. It's a relatively
complex state machine (MESI). Details vary by CPUs.

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>