From: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
To: Marcelo Tosatti <marcelo@kvack.org>
Cc: kosaki.motohiro@jp.fujitsu.com, linux-mm@kvack.org,
"Daniel Sp蚣g" <daniel.spang@gmail.com>,
"Rik van Riel" <riel@redhat.com>,
"Andrew Morton" <akpm@linux-foundation.org>
Subject: Re: [PATCH] mem notifications v3
Date: Tue, 25 Dec 2007 17:31:27 +0900 [thread overview]
Message-ID: <20071225164832.D267.KOSAKI.MOTOHIRO@jp.fujitsu.com> (raw)
In-Reply-To: <20071225122326.D25C.KOSAKI.MOTOHIRO@jp.fujitsu.com>
[-- Attachment #1: Type: text/plain, Size: 16799 bytes --]
Hi Marcelo-san
I tested your patch.
but unfortunately it doesn't works so good on large system.
description:
--------------------------------------------------------------
test machine
CPU: Itanium2 x 4
MEM: 16GB(8GB node x 2 NUMA system)
SWAP: 2GB
test program:
mem_notify_test.c
see attachement
m.sh
--------------
$ cat m.sh
#!/bin/sh
num=${1:-1}
mem=${2:-1}
echo $num $mem
for i in `seq 1 $num`; do
./mem_notify_test -m $mem &
done
--------------------------------------
1. run >10000 process test
console1# LANG=C; while [ 1 ] ;do sleep 1; date; vmstat 1 1 -S M -a; done
console2# sh m.sh 12500
Wed Dec 26 02:00:14 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
3 0 0 1561 7 12213 0 0 35 268 12 203 1 3 95 1 0
Wed Dec 26 02:00:15 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
2 0 0 602 7 13025 0 0 35 268 12 203 1 3 95 1 0
!! here 7 sec soft lockup !!
Wed Dec 26 02:00:22 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
8 1 12 42 68 13427 0 0 35 268 82 206 1 3 95 1 0
Wed Dec 26 02:00:23 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
69 0 17 38 64 13438 0 0 35 268 93 207 1 3 95 1 0
Wed Dec 26 02:00:24 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
4955 3 21 36 61 13443 0 0 35 268 103 208 1 3 95 1 0
Wed Dec 26 02:00:25 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
4138 2 28 42 57 13438 0 0 35 268 113 209 1 3 95 1 0
Wed Dec 26 02:00:26 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
5466 1 41 48 514 12975 0 0 35 269 119 211 1 3 95 1 0
Wed Dec 26 02:00:27 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
6082 0 78 38 896 12596 0 0 35 270 124 214 1 3 95 1 0
Wed Dec 26 02:00:28 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1 5 132 38 1209 12280 0 0 35 271 128 217 1 3 95 1 0
Wed Dec 26 02:00:29 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
2 0 169 38 1484 12003 0 0 35 272 132 219 1 3 95 1 0
Wed Dec 26 02:00:30 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 31 248 36 2651 10822 0 0 35 274 133 222 1 3 95 1 0
Wed Dec 26 02:00:32 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1729 3 323 42 3138 10325 0 0 35 277 134 227 1 3 95 1 0
Wed Dec 26 02:00:33 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1 2 410 50 3473 9968 0 0 35 279 134 230 1 3 95 1 0
Wed Dec 26 02:00:34 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
574 4 487 52 3737 9690 0 0 35 281 135 234 1 3 95 1 0
Wed Dec 26 02:00:36 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
5203 0 490 42 3794 9641 0 0 35 281 135 236 1 3 95 1 0
Wed Dec 26 02:00:37 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
2256 2 568 36 4025 9400 0 0 35 283 136 240 1 3 95 1 0
Wed Dec 26 02:00:38 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
2 3 631 55 4233 9167 0 0 35 285 136 243 1 3 95 1 0
Wed Dec 26 02:00:41 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
11 2 819 44 4675 8707 0 0 35 290 138 251 1 3 95 1 0
Wed Dec 26 02:00:42 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
2 4 876 48 4782 8588 0 0 35 292 138 254 1 3 95 1 0
Wed Dec 26 02:00:43 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1 6 945 36 4923 8450 0 0 35 294 139 257 1 3 95 1 0
Wed Dec 26 02:00:44 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
859 6 1001 50 5030 8317 0 0 35 296 139 260 1 3 95 1 0
Wed Dec 26 02:00:46 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1 2 1099 45 5190 8146 0 0 35 299 140 266 1 3 95 1 0
Wed Dec 26 02:00:47 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
2516 3 1190 47 5314 8009 0 0 35 301 141 270 1 3 95 1 0
Wed Dec 26 02:00:48 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
2141 3 1259 54 5406 7903 0 0 35 303 141 274 1 3 95 1 0
Wed Dec 26 02:00:49 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
3074 2 1314 44 5467 7844 0 0 35 305 141 277 1 3 95 1 0
Wed Dec 26 02:00:50 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 1 1314 45 5465 7840 0 0 35 305 142 278 1 3 95 1 0
Wed Dec 26 02:00:51 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 1313 44 5466 7840 0 0 35 305 142 278 1 3 95 1 0
!! thundering herd restoration after 30sec at swap out start !!
result: many swap out occured.
# pgrep mem_notify|wc -l
12193
about 300 process receive notify.
problem
o thundering herd occured multi times on and off.
o soft lockup occured.
o notify receive process too few.
o swap out occured
2. after test1, run file I/O
console1# LANG=C; while [ 1 ] ;do sleep 1; date; vmstat 1 1 -S M -a; done
console2# dd if=/dev/zero of=tmp bs=100M count=10
$ LANG=C; while [ 1 ] ;do sleep 1; date; vmstat 1 1 -S M -a; done
Wed Dec 26 02:21:35 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 1615 51 6048 7235 0 0 34 281 158 265 1 3 95 1 0
Wed Dec 26 02:21:36 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 1615 51 6048 7235 0 0 34 281 158 265 1 3 95 1 0
Wed Dec 26 02:21:37 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 1615 51 6048 7235 0 0 34 281 158 265 1 3 95 1 0
Wed Dec 26 02:21:38 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 1615 52 6048 7235 0 0 34 281 158 265 1 3 95 1 0
Wed Dec 26 02:21:39 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
124 6 1683 35 6174 7121 0 0 34 282 159 267 1 3 95 1 0
Wed Dec 26 02:21:40 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1262 3 1738 53 6293 6982 0 0 34 284 159 270 1 3 95 1 0
Wed Dec 26 02:21:41 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1546 1 1794 52 6404 6870 0 0 34 285 159 272 1 3 95 1 0
Wed Dec 26 02:21:42 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 5 1857 36 6525 6762 0 0 34 287 160 275 1 3 95 1 0
Wed Dec 26 02:21:43 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
3761 7 1896 35 6571 6718 0 0 34 289 160 276 1 3 95 1 0
Wed Dec 26 02:21:44 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 6 1898 43 6623 6654 0 0 34 291 160 277 1 3 95 1 0
Wed Dec 26 02:21:45 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 9 1921 36 6670 6614 0 0 34 293 160 279 1 3 95 1 0
Wed Dec 26 02:21:46 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
710 4 1944 52 6689 6582 0 0 34 294 161 280 1 3 95 1 0
Wed Dec 26 02:21:47 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 9 1958 42 6731 6549 0 0 34 295 161 281 1 3 95 1 0
Wed Dec 26 02:21:48 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 6 1978 44 6782 6498 0 0 34 297 161 284 1 3 95 1 0
!! time leap 4 sec !!
Wed Dec 26 02:21:52 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 10 2014 47 6864 6414 0 0 34 301 162 289 1 3 95 1 0
Wed Dec 26 02:21:53 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 10 2016 38 6881 6407 0 0 34 303 162 290 1 3 95 1 0
Wed Dec 26 02:21:54 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 5 2020 45 6884 6399 0 0 34 303 162 291 1 3 95 1 0
Wed Dec 26 02:21:56 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1 7 2039 43 6932 6359 0 0 34 303 162 295 1 3 95 1 0
Wed Dec 26 02:21:57 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 9 2047 36 6777 6529 0 0 34 306 162 297 1 3 95 1 0
Wed Dec 26 02:21:58 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1501 1 2047 88 6699 6569 0 0 34 307 163 301 1 3 95 1 0
Wed Dec 26 02:21:59 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 5 2047 39 6588 6733 0 0 34 307 164 302 1 3 95 1 0
Wed Dec 26 02:22:00 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 4 2047 42 6275 7035 0 0 34 308 164 303 1 3 95 1 0
Wed Dec 26 02:22:01 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 3 2047 41 6277 7036 0 0 34 310 164 303 1 3 95 1 0
Wed Dec 26 02:22:02 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 2047 42 6277 7036 0 0 34 310 164 303 1 3 95 1 0
Wed Dec 26 02:22:03 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 1 2047 44 6277 7035 0 0 34 313 164 303 1 3 95 1 0
Wed Dec 26 02:22:04 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 1 2047 46 6277 7035 0 0 34 314 164 303 1 3 95 1 0
Wed Dec 26 02:22:05 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 1 2047 46 6277 7035 0 0 34 316 164 303 1 3 95 1 0
Wed Dec 26 02:22:06 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
1 0 2047 51 6277 7035 0 0 34 316 164 302 1 3 95 1 0
Wed Dec 26 02:22:07 JST 2007
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 2047 54 6277 7035 0 0 34 316 164 302 1 3 95 1 0
some process receive notify and exit.
but too few, and it doesn't prevent swap out.
[kosaki@n3234224 mem_notify]$ pgrep mem_notify|wc -l
11992
-> about 200 process receive notify.
requirement is 1000 process(= 1GB / 1MB).
/kosaki
[-- Attachment #2: mem_notify_test.c --]
[-- Type: application/octet-stream, Size: 1773 bytes --]
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <poll.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <time.h>
#include <sys/mman.h>
#include <pthread.h>
#include <unistd.h>
#include <string.h>
#define DEFAULT_ALLOC_SIZE (1)
long alloc_size = DEFAULT_ALLOC_SIZE;
int main(int argc, char** argv)
{
struct pollfd polltable;
int fd;
int err;
void* ptr = NULL;
int c;
long size;
while((c = getopt(argc, argv, "m:")) != -1){
switch(c){
case 'm':
alloc_size = atol(optarg);
break;
default:
break;
}
}
argc -= optind;
argv += optind;
fd = open("/dev/mem_notify", O_RDONLY);
if( fd < 0 ){
perror("open ");
exit(1);
}
printf("try %ld MB\n", alloc_size);
size = alloc_size * 1024 * 1024;
ptr = mmap(0, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON|MAP_POPULATE, 0, 0);
memset(ptr, 0, size);
polltable.fd = fd;
polltable.events = POLLIN;
err = poll(&polltable, 1, -1);
if( err < 0 ){
perror("poll ");
}
if(polltable.revents){
time_t atime;
int readbuf;
err = read(fd, &readbuf, sizeof(int));
printf("read %d\n", err);
atime = time(NULL);
printf("poll ret %x %s\n", polltable.revents, ctime(&atime));
exit(1);
}
printf("mem_notify exit\n");
exit(1);
}
next prev parent reply other threads:[~2007-12-25 8:31 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-12-24 20:32 Marcelo Tosatti
2007-12-25 3:47 ` KOSAKI Motohiro
2007-12-25 4:56 ` [RFC] add poll_wait_exclusive() API KOSAKI Motohiro
2007-12-27 21:05 ` Marcelo Tosatti
2007-12-25 8:31 ` KOSAKI Motohiro [this message]
2007-12-25 10:31 ` [RFC][patch 1/2] mem notifications v3 improvement for large system KOSAKI Motohiro
2007-12-27 21:04 ` Marcelo Tosatti
2007-12-28 0:38 ` KOSAKI Motohiro
2007-12-25 10:31 ` [RFC][patch 2/2] " KOSAKI Motohiro
2007-12-25 10:41 ` KOSAKI Motohiro
2007-12-27 4:49 ` [RFC][patch] mem_notify more faster reduce load average KOSAKI Motohiro
2007-12-27 20:13 ` [PATCH] mem notifications v3 Marcelo Tosatti
2007-12-28 1:44 ` KOSAKI Motohiro
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20071225164832.D267.KOSAKI.MOTOHIRO@jp.fujitsu.com \
--to=kosaki.motohiro@jp.fujitsu.com \
--cc=akpm@linux-foundation.org \
--cc=daniel.spang@gmail.com \
--cc=linux-mm@kvack.org \
--cc=marcelo@kvack.org \
--cc=riel@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox