Summary of recent VM behavior [2.3.99-pre8]

Summary of recent VM behavior [2.3.99-pre8]

[Craig Kulesa]

Greetings...

Below are a summary of issues that I've encountered in the pre7 and pre8
kernels (at least on mid-range hardware).  I'd appreciate comments, any
enlightening information or pointers to documentation so I can answer the
questions myself. :) Also consider me a guinea pig for patches... 


1)  Unnecessary OOM situations, killing of processes
    (pathological)

Example:  On a 64 MB box, dd'ing >64 MB from /dev/zero to a file
on disk runs the kernel aground, usually killing a large RSS process 
like X11. This has been a consistent problem since pre6(-7?). This
behavior seems quite broken.  

I assume this is in the mmap code.  Cache increases as the file is written
but when the limit of physical memory is reached, problems ensue.  The CPU
is consumed ("hijacked") by kswapd or other internal kernel operations; as
though mmap'ed allocations can't be shrunk effectively (or quickly).

Not a problem w/ classzone.


2)  What's in the cache anyways?
    (puzzling)

Example: Play mp3's on an otherwise unloaded 64 MB system until cache
fills the rest of physical RAM. Then open an xterm (or GNU emacs,
or...).  After less than 10 MB of mp3 data goes goes by, close the
xterm. Open a new one. The xterm code is not in cache but is loaded from
scratch from disk, with a flurry of disk I/O (but no swapped pages). 
Why? The cache allocation is almost 50 MB -- *why* isn't it in there
somewhere?

One might imagine that the previous mp3's are solidly in cache, but
loading an mp3 only 15 MB earlier in the queue... comes from disk and not
from cache!  Why?

Another example on a 40 MB system: Open a lightweight X11/WindowMaker
session. Open Netscape 4.72 (Navigator). Close it. Log out. Login again,
load Netscape. Both X, window manager, and Netscape seem to come
straight from disk, with no swapped pages.  But the buffer cache is 
25 MB!  What's in there if the applications aren't? 

This is also seen on a 32 MB system by simply opening Navigator, closing
it, and opening it again. In kernel 2.2.xx and 2.3.99-pre5 (or with
classzone), it comes quickly out of cache.  In pre8, there's substantial
disk I/O, and about half of the pages are read from disk and not the
cache.  (??)

Before pre6 and with AA's classzone patch, a 25 MB cache seemed to contain
the "last" 25 MB of mmap'd files or I/O buffers. This doesn't seem true
anymore (?!), and it's an impediment to performance on at least
lower-end hardware.


3) Slow I/O performance

Disk access seems to incur large CPU overhead once physical memory must be
shared between "application" memory and cache.  kswapd is invoked
excessively, applications that stream data from disk hesitate, even the
mouse pointer becomes jumpy. The system load is ~50% higher in heavy disk
access than in earlier 2.2 and 2.3 kernels. 

Untarring the kernel source is a good example of this. Even a 128 MB
system doesn't do this smoothly in pre8. 

The overall memory usage in pre6 and later seems good -- there is no
gratuitous swapping as seen in pre5 (and earlier in pre2-3 etc). But the
general impression is that in the mmap code (somewhere else?), there are a
LOT of pages moved around or scanned that incurs expensive system
overhead. 

Before an "improved" means of handling vm pages (like the active/inactive
lists that Rik is working on), surely the current code in vmscan and
filemap (etc) should be shown to be fast and not conducive to this
puzzling, even pathological, behavior?  


4)  Confusion about inode_cache and dentry_cache

I'm surely confused here, but in kernel 2.3 the inode_cache and
dentry_cache are not as limited as in kernel 2.2.  Thusly,
sample applications like Redhat's 'slocate' daemon or any global use of
the "find" command will cause these slab caches to fill quickly. These
caches are effectively released under memory pressure. No problem.

But why do these "caches" show up as "used app memory" and not cache in
common tools like 'free' (or /proc/meminfo)?  This looks like a recipe for
lots of confused souls once kernel 2.4 is adopted by major distributions. 

Thoughts?


Craig Kulesa
Steward Observatory, Tucson AZ
ckulesa@as.arizona.edu

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: PATCH: Possible solution to VM problems (take 2)

[Craig Kulesa]

[Regarding Juan Quintela's wait_buffers_02.patch against pre9-2]

Wow. Much better!

The system doesn't hang itself in a CPU-thrashing-knot everytime an app
runs the used+cache allocations up to the limit of physical memory.  Cache
relinquishes gracefully, disk activity is dramatically less.  kswapd is
quiet again, whereas in pre8 it was eating 1/4 the integrated CPU time
as X11 at times.

I'm also not having the "cache content problems" I wrote about a few days
ago either.  Netscape, for example, is now perfectly content to load from
cache in 32 MB of RAM with room to spare.  General VM behavior has
pretty decent "feel" from 16 MB to 128 MB on 4 systems from 486DX2/66 to
PIII/500 under normal development load. 

In contrast, doing _anything_ while building a kernel on a 32 MB
Pentium/75 with pre8 was nothing short of a hair-pulling
experience.  [20 seconds for a bloody xterm?!]  It's smooth and
responsive now, even when assembling 40 MB RPM packages. Paging remains
gentle and not too distracting. Good. 

A stubborn problem that remains is the behavior when lots of
dirty pages pile up quickly.  Doing a giant 'dd' from /dev/zero to a
file on disk still causes gaps of unresponsiveness.  Here's a short vmstat
session on a 128 MB PIII system performing a 'dd if=/dev/zero of=dummy.dat
bs=1024k count=256':

   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  0  0   1392 100844    320  14000   0   0     0     0  186   409   0   0 100
 1  0  1   1392  53652    420  60080   0   0    12  3195  169   133   1  30  69
 0  1  1   1392  27572    444  85324   0   0     0  3487  329   495   0  18  82
 0  1  1   1392  15376    456  97128   0   0     0  3251  314   468   0   9  91
 0  1  2   1392   2332    472 109716   0   0    17  3089  308   466   0  11  89
 2  1  1   2820   2220    144 114392 380 1676   663 26644 20977 31578   0  10  90
 1  2  0   3560   2796    160 114220 284 792   303  9168 6542  7826   0  11  89
 4  2  1   3948   2824    168 114748 388 476   536 12975 9753 14203   1  11  88
 0  5  0   3944   2744    244 114496 552  88   791  4667 3827  4721   1   3  96
 2  0  0   3944   1512    416 115544  72   0   370     0  492  1417   0   3  97
 0  2  0   3916   2668    556 113800 132  36   330     9  415  1845   6   8  86
 1  0  0   3916   1876    720 114172   0   0   166     0  308  1333  14   6  80
 1  0  0   3912   2292    720 114244  76   0    19     0  347  1126   2   2  96
 2  0  0   3912   2292    720 114244   0   0     0     0  136   195   0   0 100

Guess the line when UI responsiveness was lost. :)

Yup.  Nothing abnormal happens until freemem decreases to zero, and then
the excrement hits the fan (albeit fairly briefly in this test).  After
the first wave of dirty pages are written out and the cache stabilizes,
user responsiveness seems to smooth out again. 

On the plus side...
It's relevant to note that this test caused rather reliable OOM
terminations of XFree86 from pre7-x (if not earlier) until this patch. I
haven't been able to generate any OOM process kills yet. And I've tried to
be very imaginative. :)

There's still some work needed, but Juan's patch seems to be resulting in
behavior that is clearly on the right track.  Great job guys, and thanks! 


Respectfully,
Craig Kulesa

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: PATCH: Possible solution to VM problems (take 2)

[Jan Niehusmann]

On Thu, May 18, 2000 at 03:17:25AM -0700, Craig Kulesa wrote:
> A stubborn problem that remains is the behavior when lots of
> dirty pages pile up quickly.  Doing a giant 'dd' from /dev/zero to a
> file on disk still causes gaps of unresponsiveness.  Here's a short vmstat
> session on a 128 MB PIII system performing a 'dd if=/dev/zero of=dummy.dat
> bs=1024k count=256':

While 'dd if=/dev/zero of=file' can, of course, generate dirty pages at
an insane rate, I see the same unresponsiveness when doing cp -a from 
one filesystem to another. (and even from a slow harddisk to a faster one).

Shouldn't the writing of dirty pages occur at least at the same rate 
as reading data from the slower hard disk? 

(My system: linux-2.3.99pre9-2, wait_buffers_02.patch, 
truncate_inode_pages_01.patch, lvm, PII/333Mhz, 256MB, ide & scsi hard disks)


Jan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: PATCH: Possible solution to VM problems (take 2)

[Rik van Riel]

On Thu, 18 May 2000, Jan Niehusmann wrote:
> On Thu, May 18, 2000 at 03:17:25AM -0700, Craig Kulesa wrote:
> > A stubborn problem that remains is the behavior when lots of
> > dirty pages pile up quickly.  Doing a giant 'dd' from /dev/zero to a
> > file on disk still causes gaps of unresponsiveness.  Here's a short vmstat
> > session on a 128 MB PIII system performing a 'dd if=/dev/zero of=dummy.dat
> > bs=1024k count=256':
> 
> While 'dd if=/dev/zero of=file' can, of course, generate dirty pages at
> an insane rate, I see the same unresponsiveness when doing cp -a from 
> one filesystem to another. (and even from a slow harddisk to a faster one).

I think I have this mostly figured out. I'll work on
making some small improvements over Quintela's patch
that will make the system behave decently in this
situation too.

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

Wanna talk about the kernel?  irc.openprojects.net / #kernelnewbies
http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: PATCH: Possible solution to VM problems (take 2)

[Stephen C. Tweedie]

Hi,

On Thu, May 18, 2000 at 10:41:05AM -0300, Rik van Riel wrote:
> 
> I think I have this mostly figured out. I'll work on
> making some small improvements over Quintela's patch
> that will make the system behave decently in this
> situation too.

Good, because apart from the write performance, Juan's patch seems to
work really well for the stress tests I've thrown at it so far.

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/