2.4.6pre3: kswapd dominating CPU

2.4.6pre3: kswapd dominating CPU

[ZINKEVICIUS,MATT (HP-Loveland,ex1)]

Hi gang,
For a while now 2.4 kernels have been a little flaky for us with regards to
memory management. We had chalked this up to the known VM updates going on
and have ignored and worked around it as much as we could. Now that
2.4.6pre3 is out and supposedly VM friendly and we are still seeing our
original problem I thought it was time I submitted the details to you guys
to get some help.

We are benchmarking NFS with SpecSFS 97 version 2. When the machine gets
close to running out of physical memory (according to top) kswapd quickly
become the most active process (98% CPU time). This occurs whether or not we
have any swap space enabled! The nfsd daemons get starved and our
performance drops to null. If we kill the benchmark things settle down
immediately, but we never get any memory back and afterwards if we run
anything even slightly stressful (iozone) the problem appears again
immediately. The only solution we've found is to reboot. This seems related
to whether we enable highmem in the kernel, as this problem only appears
when highmem is set to 4GB or 64GB. Any hints?

Server specs:
HP LT6000r server
4 x 700Mhz P3Xeons
4GB RAM
1GB swap partition
2.4.6pre3 kernel

Matt Zinkevicius
Modular Network Storage
Hewlett-Packard

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.4.6pre3: kswapd dominating CPU

[Rik van Riel]

On Mon, 18 Jun 2001, ZINKEVICIUS,MATT (HP-Loveland,ex1) wrote:

> For a while now 2.4 kernels have been a little flaky for us with
> regards to memory management. We had chalked this up to the
> known VM updates going on and have ignored and worked around it
> as much as we could. Now that 2.4.6pre3 is out and supposedly VM
> friendly and we are still seeing our original problem I thought
> it was time I submitted the details to you guys to get some
> help.
>
> We are benchmarking NFS with SpecSFS 97 version 2. When the
> machine gets close to running out of physical memory (according
> to top) kswapd quickly become the most active process (98% CPU
> time). This occurs whether or not we have any swap space
> enabled! The nfsd daemons get starved and our performance drops
> to null.

Ahhh, I see the problem.

The kswapd-eating-all-cpu problem is fixef in 2.4.6-pre3,
but only for DISK BASED filesystems and swap. For these
systems we will do synchronous IO once in a while and by
waiting on IO completion we avoid eating too much CPU.

For NFS we currently don't have any way to wait on IO
completion. I'll have to look into this later, I guess ;)

regards,

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.4.6pre3: kswapd dominating CPU

[Jens Axboe]

On Mon, Jun 18 2001, ZINKEVICIUS,MATT (HP-Loveland,ex1) wrote:
> [...] This seems related
> to whether we enable highmem in the kernel, as this problem only appears
> when highmem is set to 4GB or 64GB. Any hints?
> 
> Server specs:
> HP LT6000r server
> 4 x 700Mhz P3Xeons
> 4GB RAM
> 1GB swap partition
> 2.4.6pre3 kernel

WIth a machine spec'ed like that, you might want to try with the
zero-bounce patches for highmem machines. Running out of memory and
still requiring low mem bounce buffers can get ugly -- the patches won't
solve any vm issues, but they should solve the problem for you (and
boost your specsfs performance a good deal).

Haven't had time to update to 2.4.6-pre3 yet, if these don't apply let
me know:

*.kernel.org/pub/linux/kernel/people/axboe/patches/2.4.5/block-highmem-all-4.bz2

Dunno what I/O controller you used...

-- 
Jens Axboe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

RE: 2.4.6pre3: kswapd dominating CPU

[ZINKEVICIUS,MATT (HP-Loveland,ex1)]

> WIth a machine spec'ed like that, you might want to try with the
> zero-bounce patches for highmem machines. Running out of memory and
> still requiring low mem bounce buffers can get ugly -- the 
> patches won't
> solve any vm issues, but they should solve the problem for you (and
> boost your specsfs performance a good deal).
> 
> Haven't had time to update to 2.4.6-pre3 yet, if these don't apply let
> me know:
> 
> *.kernel.org/pub/linux/kernel/people/axboe/patches/2.4.5/block
> -highmem-all-4.bz2

We tried your block-highmem patch for 2.4.6pre1 (the 2.4.5 one you suggested
didn't patch cleanly). Sadly the kernel is unbootable (stops at
"uncompressing kernel..."). If you give as an updated patch for 2.4.6pre3 we
will be happy to try it!

> Dunno what I/O controller you used...

Qlogic fibre channel card (kernel's qlogicfc driver)

--Matt

PS: We also have tried Andrea's 2.4.6pre3aa2 patch. kswapd/kupdated still
runs but at much less CPU utilization (30%-70%) but for much longer periods
having an overall worsening effect. It also breaks fsync which in turn
breaks lots of things (lilo, etc).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.4.6pre3: kswapd dominating CPU

[Jens Axboe]

On Mon, Jun 25 2001, ZINKEVICIUS,MATT (HP-Loveland,ex1) wrote:
> > WIth a machine spec'ed like that, you might want to try with the
> > zero-bounce patches for highmem machines. Running out of memory and
> > still requiring low mem bounce buffers can get ugly -- the 
> > patches won't
> > solve any vm issues, but they should solve the problem for you (and
> > boost your specsfs performance a good deal).
> > 
> > Haven't had time to update to 2.4.6-pre3 yet, if these don't apply let
> > me know:
> > 
> > *.kernel.org/pub/linux/kernel/people/axboe/patches/2.4.5/block
> > -highmem-all-4.bz2
> 
> We tried your block-highmem patch for 2.4.6pre1 (the 2.4.5 one you suggested
> didn't patch cleanly). Sadly the kernel is unbootable (stops at
> "uncompressing kernel..."). If you give as an updated patch for 2.4.6pre3 we
> will be happy to try it!

I'll be updating it tomorrow anyway, it's probably the zone issue again.
At least I'll be able to verify if it is tomorrow, I'll post an update
later.

> > Dunno what I/O controller you used...
> 
> Qlogic fibre channel card (kernel's qlogicfc driver)

Ok good

-- 
Jens Axboe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/