SMP/highmem problem

SMP/highmem problem

[Ragnar Kjørstad]

I've run into a performance issue.

I'm testing SMP performance on a 4 CPU Xeon box with 8 GB RAM. 
No swap. Standard linux 2.4.4, configured with CONFIG_HIGHMEM64G
enabled.

I use a single process, bonnie++, that creates 16 1 GB files.
However, after a while, the machine gets really unresponsive (ls -l
/root takes literally several minutes when /root is not in the
cache) and the load gets really high. According to top, all CPU 
power is spent in the kernel, mainly on kswapd, bdflush and 
kupdated.

  7:19pm  up  2:11,  6 users,  load average: 8.58, 9.34, 7.34
48 processes: 42 sleeping, 5 running, 1 zombie, 0 stopped
CPU0 states:  0.1% user, 99.10% system,  0.0% nice,  0.0% idle
CPU1 states:  0.0% user, 100.0% system,  0.0% nice,  0.0% idle
CPU2 states:  0.0% user, 100.0% system,  0.0% nice,  0.0% idle
CPU3 states:  0.1% user, 99.10% system,  0.0% nice,  0.0% idle
Mem:  7721928K av, 7719308K used,    2620K free,       0K shrd,    3612K buff
Swap:       0K av,       0K used,       0K free                 7597616K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
    3 root      14   0     0    0     0 SW   99.9  0.0  56:36 kswapd
    5 root      14   0     0    0     0 RW   99.9  0.0  56:50 bdflush
    6 root      14   0     0    0     0 RW   99.9  0.0  54:44 kupdated
 1712 nobody    15   0   192  192    44 R    99.9  0.0   9:12 bonnie++
 1825 root      11   0  1064 1064   864 R     0.2  0.0   7:02 top


/proc/meminfo:
        total:    used:    free:  shared: buffers:  cached:
Mem:  3612286976 3609559040  2727936        0  3461120 3490426880
Swap:        0        0        0
MemTotal:      7721928 kB
MemFree:          2664 kB
MemShared:           0 kB
Buffers:          3380 kB
Cached:        7602924 kB
Active:          56604 kB
Inact_dirty:   3251412 kB
Inact_clean:   4298285 kB
Inact_target:    51752 kB
HighTotal:     6946808 kB
HighFree:         1048 kB
LowTotal:       775120 kB
LowFree:          1616 kB
SwapTotal:           0 kB
SwapFree:            0 kB


There are messages in the log about the kernel running out of
bounce-buffers. 

It seems related to swapping / pageing algorithms?

We've tried both 2.4.4ac6, 2.4.4ac9 and 2.4.5pre1 + Andreia's
highmem-patches. Possible the newer kernels work a little better, but
it's hard to tell because we don't have a way of actually messuring
this.

Any hints about what can be done to fix this?

This is not a production server - we're just running theese tests to see
how well linux will scale on this hardware - so, if you have any patches
you want us to try out please let us know. If you want to play around
with the machine, we can probably provide an account. Let us know.


Thanks

(please CC - I'm not subscribed to the list)



-- 
Ragnar Kjorstad
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: SMP/highmem problem

[Rik van Riel]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=X-UNKNOWN, Size: 1653 bytes --]

On Thu, 17 May 2001, [iso-8859-1] Ragnar Kjorstad wrote:

> I've run into a performance issue.

> I use a single process, bonnie++, that creates 16 1 GB files.
> However, after a while, the machine gets really unresponsive
> and the load gets really high. According to top, all CPU power
> is spent in the kernel, mainly on kswapd, bdflush and kupdated.

This is at least partly due to the following things:

1) balance_dirty_state() tests for a condition bdflush
   may not be able to resolve
2) nr_free_buffer_pages() counts free highmem pages, which
   cannot be allocated to buffer memory, as available; this
   means that bonnie++ never gets to slow down to disk speed
   and fills up all of low memory
3) because of 2) kswapd and bdflush are trying to write the
   data out to disk like crazy, but can never keep up with
   bonnie++
4) bonnie++ tries to allocate new pages all the time, but
   cannot succeed because all of low memory is full of dirty
   page cache data .. this means it loops in __alloc_pages()
   and continuously wakes up kswapd and bdflush

A few fixes for this situation have gone into 2.4.5-pre2 and
2.4.5-pre3. If you have the time, could you test if this problem
has gotten less or has gone away in the latest kernels ?

thanks,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: SMP/highmem problem

[Thomas Langås]

Rik van Riel:
> A few fixes for this situation have gone into 2.4.5-pre2 and
> 2.4.5-pre3. If you have the time, could you test if this problem
> has gotten less or has gone away in the latest kernels ?

Ok, now we've tested 2.4.5-pre3, and it's still like described before.
However, it's a bit better. 

We started bonnie++, and waited a few secs, then tested with ls'ing uncached
catalogs on /-filesystem. This was a command that took about 1m real time,
to do every time earlier (2.4.4). Now it only takes 1m real time when ls
isn't cached or hashed. This goes for every other command. However, after we
waited a few mins. the box started to become unsuable again (sshd-sessions
lagged, for instance). Killing bonnie++ fixes the situation after a little
while (1-3mins).

So, any other ideas are very welcome :)

-- 
-Thomas
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: SMP/highmem problem

[Rik van Riel]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=X-UNKNOWN, Size: 1527 bytes --]

On Sat, 19 May 2001, [iso-8859-1] Thomas Langas wrote:
> Rik van Riel:
> > A few fixes for this situation have gone into 2.4.5-pre2 and
> > 2.4.5-pre3. If you have the time, could you test if this problem
> > has gotten less or has gone away in the latest kernels ?
> 
> Ok, now we've tested 2.4.5-pre3, and it's still like described before.
> However, it's a bit better. 

Whooops, now that I looked at the source code for -pre3
I realise the particular patch which could fix this
problem hasn't gone into -pre3.

I'll send a patch SOON (almost like the one I sent a
few days ago, but with a few new fixes which have been
accumulating in the last few days).

> So, any other ideas are very welcome :)

The basis for the patch will be the page_alloc.c VM
patch on http://www.surriel.com/patches/, but with 2
minor changes:

1) don't allow GFP_BUFFER pages to loop in __alloc_pages(),
   but have them fail after a while ... needed to avoid
   deadlocks
2) never allow nr_free_buffer_pages to return a number
   larger than how many dirty pages would reasonably fit
   in ZONE_DMA and ZONE_NORMAL ... should fix your problem

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: SMP/highmem problem

[Thomas Langås]

Rik van Riel:
> > So, any other ideas are very welcome :)
> 
> The basis for the patch will be the page_alloc.c VM
> patch on http://www.surriel.com/patches/, but with 2
> minor changes:
> 
> 1) don't allow GFP_BUFFER pages to loop in __alloc_pages(),
>    but have them fail after a while ... needed to avoid
>    deadlocks

I added a counter which didn't "goto try_again" if it hit 10.

> 2) never allow nr_free_buffer_pages to return a number
>    larger than how many dirty pages would reasonably fit
>    in ZONE_DMA and ZONE_NORMAL ... should fix your problem

Made this function do return MIN(sum, ZONE_DMA+ZONE_NORMAL)

This did actually improve things _alot_ :)  We're going to perform some more
tests sometime tomorrow, but as far as I can tell, things are going way
better now than they did before. However, it might not be perfect yet...

-- 
-Thomas
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/