Running out of vmalloc space

Running out of vmalloc space

[David Pinedo]

Hello.  My name is David Pinedo and I subscribed to this list a few days
ago.  I work for Hewlett-Packard, and am in the process of porting the
drivers for the FX10 graphics boards from Red Hat 6.2 to Red Hat 7.1.
Our customers are begging us to support the FX10 on RH7.1, primarily
because of the large memory capabilities in the 2.4.2 kernel.
 
I only have minimal knowledge of the Linux kernel, enough to be able to
create the kernel module driver for the FX10.  My apologies for jumping
into this email list and the inevitable newbie mistakes I may make.  I
have a strong business need to get these graphics boards working
correctly in the 2.4.2 kernel, and I think this email list may be the
only place I can get some help. If I should be using some other forum
for my questions, I would appreciate if someone would point me to it.
 
Porting the driver to 2.4.2 was not terribly difficult.  Most of the
changes were in code that translates to and from physical addresses and
virtual addresses.  There were a few changes I had to make due to
difference in the gcc compiler on RH7.1 vs RH6.2.
 
The FX10 has a very large frame buffer and control space (the control
space is where the registers to control the device reside).  The frame
buffer is 16Mbytes and the control space is 32Mbytes.  The address space
for the frame buffer and control space is allocated from the kernel vm
address space, using get_vm_area() in mm/vmalloc.c.
 
On Linux, HP supports up to two FX10 boards in the system.  In order to
use two FX10 boards, the kernel driver needs to map the frame buffer and
control space for both of the boards.  That's a lot of address space,
2*(16M+32M)=96M to be exact.  Using this much virtual address space on a
stock RH7.1 smp kernel on a system with 0.5G of memory didn't seem to
be a problem.  However, a colleague reported a problem to me on his
system with 1.0G of memory -- the X server was exiting with an error
message indicating that it couldn't map both devices.
 
On investigating the problem, I found that a call to get_vm_area was
failing because the kernel was running out of vmalloc space.  It seems
that the vmalloc space is smaller when more memory was installed on the
system:
 
                        .5G RAM           1.0G RAM
                       ----------        ---------
        VMALLOC_END    0xfdffe000        0xfdffe000
        VMALLOC_START  0xe0800000        0xf8800000
                       ----------        ---------
        space avail    0x1d7fe000(471M)  0x057FE000(87M)
 
 
I found that if I reconfigure the kernel with Maximum Virtual Memory set
to 2G (sets CONFIG_2GB), the vmalloc space is larger and the problem
goes away.  I couldn't quite figure out what the implications of
changing Maximum Virtual Memory really are.  The Help button when using
"make xconfig" says there is no help available.  Could someone enlighten
me?  Will this fix also work when I add more memory to the system?
 
Another method of fixing this problem that seems to work is to change
the constant VMALLOC_RESERVE in arch/i386/kernel/setup.c.  I changed the
line that defines it from:
 
   #define VMALLOC_RESERVE (unsigned long)(128 << 20)
 
to:
 
   #define VMALLOC_RESERVE (unsigned long)(256 << 20)
 
What are the implications of making such a change?  Will it work when
there is less or more memory in the system?  Should this be a
configurable kernel parameter?
 
Thanks for any information anyone can provide.
 
David Pinedo
Hewlett-Packard Company
Fort Collins, Colorado
dp@fc.hp.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Stephen C. Tweedie]

Hi,

On Thu, May 17, 2001 at 11:13:00AM -0600, David Pinedo wrote:

> On Linux, HP supports up to two FX10 boards in the system.  In order to
> use two FX10 boards, the kernel driver needs to map the frame buffer and
> control space for both of the boards.  That's a lot of address space,
> 2*(16M+32M)=96M to be exact.  Using this much virtual address space on a
> stock RH7.1 smp kernel on a system with 0.5G of memory didn't seem to
> be a problem.  However, a colleague reported a problem to me on his
> system with 1.0G of memory -- the X server was exiting with an error
> message indicating that it couldn't map both devices.

You obviously want to be able to map this memory into the X server's
virtual address space, but do you really need to map it into the
kernel's VA too?  

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Matti Aarnio]

On Thu, May 17, 2001 at 11:13:00AM -0600, David Pinedo wrote:
[ Why vmalloc() space is so small ? ]

  Hua Ji summarized quite well what the kernel does, and where.

  There are 32bit machines which *can* access whole 4G kernel space
  separate from simultaneous 4G user space, however i386 is not one
  of those.

  PAE36 doesn't help either -- aside of the PHYSICAL memory addressability
  increase, the problem is in 4G choke point in address calculations which
  causes  32-bit segment register value be added on  32-bits address, but
  only for the low 32 bits, loosing "up-shifted" top 4 bits.  The mapping
  tables will then expand that 32-bit result to 36 bits of PAE.

  If you can come up with some magic instruction which does data move
  from/to alternate memory mapping context than what is currently running
  (e.g. userspace or kernel), preferrably privileged instruction, a LOT
  of people would be very glad -- and very nearly overnight we could 
  supply 4G space for both user and kernel spaces.

  In Motorola 68k series there is such a thing, called 'movec'.

> Thanks for any information anyone can provide.
>  
> David Pinedo
> Hewlett-Packard Company
> Fort Collins, Colorado
> dp@fc.hp.com

/Matti Aarnio  -- who much prefers clean 64-bit pointers...
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Christoph Hellwig]

On Thu, May 17, 2001 at 10:16:10PM +0300, Matti Aarnio wrote:
> On Thu, May 17, 2001 at 11:13:00AM -0600, David Pinedo wrote:
> [ Why vmalloc() space is so small ? ]
> 
>   Hua Ji summarized quite well what the kernel does, and where.
> 
>   There are 32bit machines which *can* access whole 4G kernel space
>   separate from simultaneous 4G user space, however i386 is not one
>   of those.

Kanoj Sarcar has written a patch for Linux 2.2 to allow exactly this.
Take a look at http://oss.sgi.com/projects/bigmem/, the page also
contains a nice explanation of what the changes actually do.

	Christoph

-- 
Whip me.  Beat me.  Make me maintain AIX.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Matti Aarnio]

On Thu, May 17, 2001 at 09:23:10PM +0200, Christoph Hellwig wrote:
> > On Thu, May 17, 2001 at 11:13:00AM -0600, David Pinedo wrote:
> > [ Why vmalloc() space is so small ? ]
> > 
> >   Hua Ji summarized quite well what the kernel does, and where.
> > 
> >   There are 32bit machines which *can* access whole 4G kernel space
> >   separate from simultaneous 4G user space, however i386 is not one
> >   of those.
> 
> Kanoj Sarcar has written a patch for Linux 2.2 to allow exactly this.
> Take a look at http://oss.sgi.com/projects/bigmem/, the page also
> contains a nice explanation of what the changes actually do.

   It doesn't supply separate SIMULTANEOUS address spaces.
   It does pageing table juggling by having some 0.2 GB always
   mapped, and then switching things back and forth at tables.

   Kanoj's approach gives larger address spaces for both spaces
   (user, and kernel), but it can't circumvent i386 hardware
   limitations.

   And like Kanoj notes, there are performance penalities.
   If a way can be found which shows none of those penalties,
   Linus would accept the code, I believe.

> 	Christoph

/Matti Aarnio
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[David Pinedo]

"Stephen C. Tweedie" wrote:
> 
> Hi,
> 
> On Thu, May 17, 2001 at 11:13:00AM -0600, David Pinedo wrote:
> 
> > On Linux, HP supports up to two FX10 boards in the system.  In order to
> > use two FX10 boards, the kernel driver needs to map the frame buffer and
> > control space for both of the boards.  That's a lot of address space,
> > 2*(16M+32M)=96M to be exact.  Using this much virtual address space on a
> > stock RH7.1 smp kernel on a system with 0.5G of memory didn't seem to
> > be a problem.  However, a colleague reported a problem to me on his
> > system with 1.0G of memory -- the X server was exiting with an error
> > message indicating that it couldn't map both devices.
> 
> You obviously want to be able to map this memory into the X server's
> virtual address space, but do you really need to map it into the
> kernel's VA too?
> 
> Cheers,
>  Stephen


Unfortunately, yes. It has to be in the kernel's virtual address space,
because the kernel graphics driver initiates DMAs to and from the
graphics board, which can only be done from the kernel using locked down
physical memory.

Thanks for all the replies to the questions I raised.

David Pinedo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Andi Kleen]

On Fri, May 18, 2001 at 12:48:38AM +0200, David Pinedo wrote:
> Unfortunately, yes. It has to be in the kernel's virtual address space,
> because the kernel graphics driver initiates DMAs to and from the
> graphics board, which can only be done from the kernel using locked down
> physical memory.

If it doesn't do any direct CPU accesses to the graphics board you could
do the DMA even without having the board mapped in the MMU. You just have
to work with physical addresses.

-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Stephen C. Tweedie]

Hi,

On Thu, May 17, 2001 at 04:48:38PM -0600, David Pinedo wrote:

> Unfortunately, yes. It has to be in the kernel's virtual address space,
> because the kernel graphics driver initiates DMAs to and from the
> graphics board, which can only be done from the kernel using locked down
> physical memory.

Why does it have to be virtually contiguous?  If you are using vmalloc
then you are necessarily using physically-discontiguous space.  If you
are doing DMA on that space then the kernel isn't accessing it
virtually at all, except perhaps to populate it, which can be done
trivially page by page without having the space virtually contiguous.

It is often a lot easier on the kernel programmer if the addresses
are virtually contiguous, but it is very rarely necessary.  It is
trivial to create an "offset_to_virt" helper function which translates
an offset within one of your pci regions to a virtual kernel address
by indexing a physical page location array which contains the list of
allocated pages.  The *only* thing which is measurably more difficult
without vmalloc is crossing page boundaries.

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Christoph Hellwig]

On Thu, May 17, 2001 at 04:48:38PM -0600, David Pinedo wrote:
> Unfortunately, yes. It has to be in the kernel's virtual address space,
> because the kernel graphics driver initiates DMAs to and from the
> graphics board, which can only be done from the kernel using locked down
> physical memory.

Take a look at drivers/char/raw.c on how to lock down userpages.

	Christoph

-- 
Of course it doesn't work. We've performed a software upgrade.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[David Pinedo]

I followed up on the suggestion of several folks to not map the graphics
board into kernel vm space. While investigating how to do that, I
discovered that the frame buffer space did not need to be mapped -- it
was already being mapped with the control space. So instead of needing
(32M+16M)*2=96M of vmalloc space, I only need 32M*2=64M. That change
seemed easier than figuring out how not to map the board into kernel vm
space, so...
 
Given that VMALLOC_RESERVE is 128M, and there was about 92M left when my
graphics driver was being initialized, I thought I should have plenty of
room to map the two graphics boards. It worked -- the first time. If I
exitted the X server and restarted it, I would get the same errors from
the X server as when it was not able to map one of the devices.

I verified that the kernel vm space was being freed when the X server is
shut down, so that wasn't the problem. On further investigation, I found
that after the X server initialized the graphics boards, somebody else
was allocating a 24k chunk of vm space, immiediately after the two
chunks allocated by the graphics driver. When the graphics driver tried
to re-allocate those two chunks the next time the X server was started,
it was able to get the first, but not the second chunk. I looked at the
get_vm_area code in vmalloc.c, I found that it will not allow a vm space
to be allocated from a free space that is exactly the size of the space
being allocated. I think the statement:

    if (size + addr < (unsigned long) tmp->addr)
          break;       

should be:  

    if (size + addr <= (unsigned long) tmp->addr)
          break;

Making this change seems to fix my problem. :-)

David Pinedo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Stephen C. Tweedie]

Hi,

On Tue, May 22, 2001 at 05:15:26PM -0600, David Pinedo wrote:
> I followed up on the suggestion of several folks to not map the graphics
> board into kernel vm space. While investigating how to do that, I
> discovered that the frame buffer space did not need to be mapped -- it
> was already being mapped with the control space. So instead of needing
> (32M+16M)*2=96M of vmalloc space, I only need 32M*2=64M. That change
> seemed easier than figuring out how not to map the board into kernel vm
> space, so...

...so you'll end up with a driver which will work fine as long as
nobody tries to load it in parallel with another driver which tries to
pull the same stunt.  It's an easy way out which doesn't work if
everybody takes the same easy way out.

I *really* think you need to be avoiding the mapping in the first
place if at all possible.

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[David Pinedo]

"Stephen C. Tweedie" wrote:
> 
> Hi,
> 
> On Tue, May 22, 2001 at 05:15:26PM -0600, David Pinedo wrote:
> > I followed up on the suggestion of several folks to not map the graphics
> > board into kernel vm space. While investigating how to do that, I
> > discovered that the frame buffer space did not need to be mapped -- it
> > was already being mapped with the control space. So instead of needing
> > (32M+16M)*2=96M of vmalloc space, I only need 32M*2=64M. That change
> > seemed easier than figuring out how not to map the board into kernel vm
> > space, so...
> 
> ...so you'll end up with a driver which will work fine as long as
> nobody tries to load it in parallel with another driver which tries to
> pull the same stunt.  It's an easy way out which doesn't work if
> everybody takes the same easy way out.
> 
> I *really* think you need to be avoiding the mapping in the first
> place if at all possible.
> 
> Cheers,
>  Stephen


The graphics kernel driver needs to access the device, so it needs to be
mapped into kernel vm space. I might be able to get away with not
mapping all of the device. For example, the driver only accesses a
subset of the registers on the graphics board, and doesn't access the
framebuffer. I have a customer waiting for other bug fixes, so such a
change would have to take a lower priority.

I could easily imagine a scenario on a future graphics device where the
kernel driver accesses a large percentage of the address space of the
device, so the demand on kernel vm memory space would be high. As
framebuffers get larger in the future, the need for kernel vm space will
also increase.

David P
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Stephen C. Tweedie]

Hi,

On Wed, May 23, 2001 at 10:14:12AM -0600, David Pinedo wrote:

> The graphics kernel driver needs to access the device, so it needs to be
> mapped into kernel vm space. I might be able to get away with not
> mapping all of the device. For example, the driver only accesses a
> subset of the registers on the graphics board, and doesn't access the
> framebuffer. I have a customer waiting for other bug fixes, so such a
> change would have to take a lower priority.

That's the core of the issue.  I can't see why you need direct access
to all 64MB of device address space.  Doing an ioremap() of the
relevant register space is fine.  It's what PCI device drivers are
expected to do, and it will cooperate with other drivers.  Mapping the
entire framebuffer is excessive.

> I could easily imagine a scenario on a future graphics device where the
> kernel driver accesses a large percentage of the address space of the
> device, so the demand on kernel vm memory space would be high. As
> framebuffers get larger in the future, the need for kernel vm space will
> also increase.

I remain to be convinced.  WHY does the kernel need to access the
framebuffer at all?  The X server can map the entire frame
buffer into its own VA without it being in the kernel.  The kernel can
set up DMA into the framebuffer without having that memory mapped
*anywhere*.  It can also talk to the acceleration engine without
having the framebuffer mapped.  I can't see a justification for having
the whole framebuffer kernel-mapped here.

Indeed, the X server is sufficiently performance-critical that you
usually want to avoid the kernel accessing the framebuffer at all: far
better to do it in the X server and avoid any protection switches into
the kernel domain.  

Cheers,
 Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Andrew Morton]

David Pinedo wrote:
>
>     if (size + addr < (unsigned long) tmp->addr)
>           break;
> 
> should be:
> 
>     if (size + addr <= (unsigned long) tmp->addr)
>           break;
> 
> Making this change seems to fix my problem. :-)

For the record - I sent this change on to Linus and
he applied it, incorrectly attributed to myself :(
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: Running out of vmalloc space

[Matti Aarnio]

On Thu, May 17, 2001 at 02:58:29PM -0700, Hua Ji wrote:
> FYI:
>   http://www.linux-mm.org/more_than_1GB.shtml
> The above url gives a good introduction for what we are discussing. 

No.  That is repeating Kanoj's patch logic.  What got into 2.4 kernel
is slightly different.

But like Stephen wondered, is there truly need to map the cards entirely
into KERNEL space at all ?    It is fairly trivial to create mmap() driver
function to map them to user-space process.

In display drivers I can understand a need to map parts (control registers,
including command pipeline insert points) of the cards to kernel, but not
all of the buffer spaces -- at least not all the time.

/Matti Aarnio
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

RE: Running out of vmalloc space

[Hua Ji]

http://www.linux-mm.org/more_than_1GB.shtml

The above url gives a good introduction for what we are discussing. 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Running out of vmalloc space

[Andi Kleen]

On Thu, May 17, 2001 at 08:51:49PM +0200, Hua Ji wrote:
> For example, if your machine has a physical memory of 256M. And then your
> vmalloc can only manage
> (1G-256M-8M) space.

Not quite true with 2.4 anymore: Linux can put physical memory that doesn't
fit between PAGE_OFFSET and the beginning of special mappings into highmem,
where it is only mapped from on demand.
Highmem has some penalties (double buffering on IO, cannot be used directly
by many kernel subsystems), but is still usable.

So moderately increasing the vmalloc area should not be a big problem. Of 
course it should still leave some directly mapped space for the kernel.

-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

RE: Running out of vmalloc space

[Hua Ji]

>What are the implications of making such a change?  Will it work when
>there is less or more memory in the system?  Should this be a
>configurable kernel parameter?

My 2 cents:

By default, linux kernel space starts from PAGE_OFFSET, which is 0xC0000000.
In other words,
All the kernel can only have 1G memory left for usage, if/when under a 32bit
CPU.

Even with the 1G left, the cake left for vmalloc is much less than the 1G.
Kernel
will map the PAGE_OFFSET~PAGE_OFFSET+physical_memory for kmalloc usage. The
real start point for
vmalloc is high_memory + 8M(this is a hole). 

Hence, we can understand that the virtual address left for vmalloc is really
small.

For example, if your machine has a physical memory of 256M. And then your
vmalloc can only manage
(1G-256M-8M) space.

If we go through the get_vma_area that is called by vmalloc(), we will find
this:

------------------------------------------
addr = VMALLOC_START;
.....

 if (addr > VMALLOC_END-size) {	 
			kfree(area);
			return NULL;
		}
------------------------------------------

Therefore, it is very possible that your driver codes can't find **big
enough hole** in the vmlist, which
is a global linked list for maintaining all the vm_struct data structures.

For enlarging the managed memory, you can try this:

* change the PAGE_OFFSET to 0x80000000(for example) from 0xC000000. Then you
will have 1G extra memory managable:-). However, the side effect is: your
user level tasks can only range from 0x0 to 0x8000000(2G).



Wish helpful,

Mike
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/