map_user_kiobuf and 1 Gb (2.4-test8)

map_user_kiobuf and 1 Gb (2.4-test8)

[Steve Case]

I'm working on a device driver module for our PCI interface cards which
attempts to map user memory for DMA. I was pleased to find the
map_user_kiobuf function and its allies, since this appears to do
exactly what I need. Everything worked fine, until I sent it to a
customer who has a system w/  1 Gb of memory - it locked up real good as
soon as he tried DMA. After making sure we had the same software -
2.4-test8, with the CONFIG_HIGHMEM4G flag set, two pentium IIIs, etc. we
discovered that everything worked if he pulled a DIMM and went to 768M.
The actual amount of memory used by his test remained fairly small.

In the driver I use map_user_iobuf with the user space address, then
cycle through the maplist filling in a scatter-gather list:

/* scatter - gather list */
struct {
    u_int addr;
    u_int size;
} sg;

size=0;
while (size < xfersize)
    {

                 sg.addr =
virt_to_bus(page_address(iobuf.maplist[entrys]));

/* deal with page crossings */

                 if ((u_int)sg.addr & (PAGE_SIZE - 1))
                    thissize = PAGE_SIZE - ((u_int)sg.addr & (PAGE_SIZE
- 1)) ;
                else
                    thissize= PAGE_SIZE;

                if (size + thissize > xfersize)
                        thissize = xfersize - size ;

/* set scatter-gather element size */
                   sg.size = thissize;

                    size += thissize;

    }

The scatter-gather list itself is allocated using kmalloc(); the bus
address is retrieved using virt_to_bus(). We present our card with the
bus address of the scatter-gather list, from which it does DMA to get
the address/size pairs. This works fine for < 1Gb. So, either the
map_user_iobuf function is giving me a bad (unmapped) address, or
kmalloc/virt_to_bus is breaking down at 1Gb.

Are there any obvious gotchas about using kiobuf in systems >= 1 GB?

Thanks,

Steve Case
Engineering Design Team
steve@edt.com


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: map_user_kiobuf and 1 Gb (2.4-test8)

[Stephen Tweedie]

Hi,

On Thu, Oct 05, 2000 at 01:56:09PM -0700, Steve Case wrote:
> I'm working on a device driver module for our PCI interface cards which
> attempts to map user memory for DMA. I was pleased to find the
> map_user_kiobuf function and its allies, since this appears to do
> exactly what I need. Everything worked fine, until I sent it to a
> customer who has a system w/  1 Gb of memory - it locked up real good as
> soon as he tried DMA

>       sg.addr = virt_to_bus(page_address(iobuf.maplist[entrys]));

>From include/asm-i386/pgtable.h:

  /*
   * Permanent address of a page. Obviously must never be
   * called on a highmem page.
   */
  #define page_address(page) ((page)->virtual)

The 2.4 kernel is able to deal with >=1GB physical memory in its VM,
but the IO subsystem on i386 architectures is still not ready for such
memory.  What happens today is that network IO is restricted to low
memory (roughly speaking, below the 900MB mark) by simply ensuring
that skbuff packet buffers are always allocated from low memory in the
first place, and that disk IO is restricted to low memory by doing
"bounce buffer" operations --- if you attempt disk IO to a high memory
page, the kernel will allocate a temporary low-memory page for the IO
and will copy the IO results to/from high memory as required.

The easiest way to deal with this for now will be to special-case high
memory pages with "if (PageHighMem(page)) {}" in a similar manner.
You'll probably see much better support for IO to high mem pages in
2.5, but in 2.4 doing a copy is the safest way to go.

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/