Re: [linux-usb-devel] Patch for UHCI driver (from kernel 2.6.6).

Re: [linux-usb-devel] Patch for UHCI driver (from kernel 2.6.6).

[Alan Stern]

On Mon, 14 Jun 2004, David Brownell wrote:

> Seems like the dma_alloc_coherent() API spec can't be
> implemented on such machines then, since it's defined
> to return memory(*) such that:
> 
>    ... a write by either the device or the processor
>    can immediately be read by the processor or device
>    without having to worry about caching effects.
> 
> Seems like the documentation should change to explain
> under what circumstances "coherent" memory will exhibit
> cache-incoherent behavior, and how to cope with that.
> (Then lots of drivers would need to change.)
> 
> OR ... maybe the bug is just that those PPC processors
> can't/shouldn't claim to implement that API.  At which
> point all drivers relying on that API (including all
> the USB HCDs and many of the USB drivers) stop working.
> 
> - Dave
>
> (*) DMA-API.txt uses two terms for this:  "coherent" and "consistent".
>      DMA-mapping.txt only uses "consistent".

That text strikes me as rather ambiguous.  Maybe it's intended to mean
that a write by either side can be read immediately by the other side, and
the values read will be the ones written (i.e., the read won't get stale
data from some cache).  It doesn't specify what happens to the other data
bytes in the same cache line which _weren't_ written -- maybe they'll be
messed up.

In other words, with "coherent" or "consistent" memory (there is some
technical distinction between the two terms but I don't know what it is)  
you don't have to worry about reading stale data from a cache, but you
might still have to worry about data unintentionally being overwritten
with garbage.

Clearly this is not a tremendously useful guarantee, but I guess it's 
better than nothing.

Maybe someone on linux-mm can clarify things for the rest of us.  For
anyone interested, this thread started with:

http://marc.theaimsgroup.com/?l=linux-usb-devel&m=108728413809788&w=2

Alan Stern


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [linux-usb-devel] Patch for UHCI driver (from kernel 2.6.6).

[David Brownell]

Alan Stern wrote:
> On Mon, 14 Jun 2004, David Brownell wrote:
> 
> 
>>Seems like the dma_alloc_coherent() API spec can't be
>>implemented on such machines then, since it's defined
>>to return memory(*) such that:
>>
>>   ... a write by either the device or the processor
>>   can immediately be read by the processor or device
>>   without having to worry about caching effects.
>>
>>...
> 
> That text strikes me as rather ambiguous.  ...
> ...  It doesn't specify what happens to the other data
> bytes in the same cache line which _weren't_ written -- maybe they'll be
> messed up.

Actually I thought it was quite explicit:  "without having
to worry about caching effects".  What you described is
clearly a caching effect:  caused by caching.  And maybe
fixable by appropriate cache-flushing, or other cache-aware
driver logic ... making it "worry about caching effects".
Like the patch from Nicolas.

Maybe what we really need is patches to make USB switch to
dma_alloc_noncoherent(), checking dma_is_consistent() to
see whether a given QH/TD/ED/iTD/sITD/FSTN/... needs to be
explicitly flushed from cpu cache before handover to HC.

- Dave



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [linux-usb-devel] Patch for UHCI driver (from kernel 2.6.6).

[Duncan Sands]

> Actually I thought it was quite explicit:  "without having
> to worry about caching effects".  What you described is
> clearly a caching effect:  caused by caching.

Is it really a cache effect?  Isn't it caused by the hc writing
more bytes to memory than you expected?  Now, it so
happens that the number of bytes it writes is equal to the
cache line size (or so it seems), but isn't that irrelevant?

Ciao,

Duncan.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [linux-usb-devel] Patch for UHCI driver (from kernel 2.6.6).

[Alan Stern]

On Tue, 15 Jun 2004, David Brownell wrote:

> Alan Stern wrote:
> > On Mon, 14 Jun 2004, David Brownell wrote:
> > 
> > 
> >>Seems like the dma_alloc_coherent() API spec can't be
> >>implemented on such machines then, since it's defined
> >>to return memory(*) such that:
> >>
> >>   ... a write by either the device or the processor
> >>   can immediately be read by the processor or device
> >>   without having to worry about caching effects.
> >>
> >>...
> > 
> > That text strikes me as rather ambiguous.  ...
> > ...  It doesn't specify what happens to the other data
> > bytes in the same cache line which _weren't_ written -- maybe they'll be
> > messed up.
> 
> Actually I thought it was quite explicit:  "without having
> to worry about caching effects".  What you described is
> clearly a caching effect:  caused by caching.  And maybe
> fixable by appropriate cache-flushing, or other cache-aware
> driver logic ... making it "worry about caching effects".
> Like the patch from Nicolas.

No, you misunderstood what I wrote and misinterpreted the text.  The text 
says:

	... a write by either the device or the processor
	can immediately be read by the processor or device 
	without having to worry about caching effects.

This means that when you read _the data that was written_ you don't have 
to worry about caching effects.

This does _not_ mean that when you read _other data stored nearby_ you
don't have to worry about caching effects.

That is the problem Nicolas wants to solve.  According to him, the problem 
occurs not when reading the data stored by the device but when reading 
data stored nearby by the CPU and mangled during the DMA transfer.

Now maybe the intention behind "consistent" or "coherent" mappings is that 
the memory really will behave as one would naively expect, and a write to 
one portion of a cache line won't mess up the contents of the remainder.  
But the text doesn't say this.  That's why I said it is ambiguous.

> Maybe what we really need is patches to make USB switch to
> dma_alloc_noncoherent(), checking dma_is_consistent() to
> see whether a given QH/TD/ED/iTD/sITD/FSTN/... needs to be
> explicitly flushed from cpu cache before handover to HC.

Will flushing the CPU cache really solve these problems?  If the hardware
that handles the DMA transfer always writes an entire cache line, and if
it doesn't read the old contents before doing a partial write, then data
stored in the same cache line as a DMA buffer is subject to overwriting
whether it has been flushed or not.

Alan Stern

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [linux-usb-devel] Patch for UHCI driver (from kernel 2.6.6).

[David Brownell]

>>>That text strikes me as rather ambiguous.  ...
>>>...  It doesn't specify what happens to the other data
>>>bytes in the same cache line which _weren't_ written -- maybe they'll be
>>>messed up.
>>
>>Actually I thought it was quite explicit:  "without having
>>to worry about caching effects".  What you described is
>>clearly a caching effect:  caused by caching.  And maybe
>>fixable by appropriate cache-flushing, or other cache-aware
>>driver logic ... making it "worry about caching effects".
>>Like the patch from Nicolas.
> 
> 
> No, you misunderstood what I wrote and misinterpreted the text.  The text 
> says:
> 
> 	... a write by either the device or the processor
> 	can immediately be read by the processor or device 
> 	without having to worry about caching effects.
> 
> This means that when you read _the data that was written_ you don't have 
> to worry about caching effects.

It doesn't limit the "without having to worry" to just the bytes
written.  And the rest of that API spec doesn't even suggest that
there might be an issue there.  I think you're trying to read
things into that text that aren't there.

On the other hand, see dma_alloc_noncoherent() ... I hope you'll
agree that the specification for "noncoherent" memory clearly
expects those caching effects to exist.  It specifically says
that you need to understand cache line sharing issues to use the
API correctly, and even exposes the cache line size.

It seems most likely to me that this particular PPC hardware
can't implement dma_alloc_coherent(), and the code implementing
that routine should be called dma_alloc_noncoherent() instead.



>>Maybe what we really need is patches to make USB switch to
>>dma_alloc_noncoherent(), checking dma_is_consistent() to
>>see whether a given QH/TD/ED/iTD/sITD/FSTN/... needs to be
>>explicitly flushed from cpu cache before handover to HC.
> 
> 
> Will flushing the CPU cache really solve these problems?  If the hardware

That's why we need PPC expertise applied here.  Nicolas didn't
seem to have answers for all the questions I was asking.


> that handles the DMA transfer always writes an entire cache line, and if
> it doesn't read the old contents before doing a partial write, then data
> stored in the same cache line as a DMA buffer is subject to overwriting
> whether it has been flushed or not.

Addressed by using dma_alloc_noncoherent() and dma_cache_sync(); yes?

- Dave




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [linux-usb-devel] Patch for UHCI driver (from kernel 2.6.6).

[Alan Stern]

On Tue, 15 Jun 2004, David Brownell wrote:

> > 	... a write by either the device or the processor
> > 	can immediately be read by the processor or device 
> > 	without having to worry about caching effects.
> > 
> > This means that when you read _the data that was written_ you don't have 
> > to worry about caching effects.
> 
> It doesn't limit the "without having to worry" to just the bytes
> written.

It's hard to say exactly what it means because it's ungrammatical.  "A 
write ... can immediately be read..." -- what does it mean to read a 
write?

>  And the rest of that API spec doesn't even suggest that
> there might be an issue there.  I think you're trying to read
> things into that text that aren't there.

Maybe so.  It's not worth spending more time on this point, anyhow.

Alan Stern

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [linux-usb-devel] Patch for UHCI driver (from kernel 2.6.6).

[Guennadi Liakhovetski]

On Tue, 15 Jun 2004, Alan Stern wrote:

> On Mon, 14 Jun 2004, David Brownell wrote:
>
>> Seems like the dma_alloc_coherent() API spec can't be
>> implemented on such machines then, since it's defined
>> to return memory(*) such that:
>>
>>    ... a write by either the device or the processor
>>    can immediately be read by the processor or device
>>    without having to worry about caching effects.
>>
>> Seems like the documentation should change to explain
>> under what circumstances "coherent" memory will exhibit
>> cache-incoherent behavior, and how to cope with that.
>> (Then lots of drivers would need to change.)
>>
>> OR ... maybe the bug is just that those PPC processors
>> can't/shouldn't claim to implement that API.  At which
>> point all drivers relying on that API (including all
>> the USB HCDs and many of the USB drivers) stop working.
>>
>> - Dave
>>
>> (*) DMA-API.txt uses two terms for this:  "coherent" and "consistent".
>>      DMA-mapping.txt only uses "consistent".
>
> That text strikes me as rather ambiguous.  Maybe it's intended to mean
> that a write by either side can be read immediately by the other side, and
> the values read will be the ones written (i.e., the read won't get stale
> data from some cache).  It doesn't specify what happens to the other data
> bytes in the same cache line which _weren't_ written -- maybe they'll be
> messed up.
>
> In other words, with "coherent" or "consistent" memory (there is some
> technical distinction between the two terms but I don't know what it is)

<quote>