From: gaoqiang <gaoqiangscut@gmail.com>
To: Sha Zhengju <handai.szj@gmail.com>
Cc: Michal Hocko <mhocko@suse.cz>,
cgroups@vger.kernel.org, linux-mm@kvack.org
Subject: Re: question about memsw of memory cgroup-subsystem
Date: Tue, 17 Apr 2012 16:18:17 +0800 [thread overview]
Message-ID: <op.wcwhcri3n27o5l@gaoqiang-d1.corp.qihoo.net> (raw)
In-Reply-To: <4F8CE2A6.7070004@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2507 bytes --]
在 Tue, 17 Apr 2012 11:25:26 +0800，Sha Zhengju <handai.szj@gmail.com>
写道:
the vmware machine has about 2G swap space,which should be quite enough..
Days ago I can produce it on a physical machine, not any more now.
the /var/log/message- was still there (the attchment),so I think
it was not a mistake.
I tried on my laptop,with the same system, easy to reproduce..
> On 04/16/2012 11:43 AM, gaoqiang wrote:
>> 在 Fri, 13 Apr 2012 22:49:54 +0800，Michal Hocko <mhocko@suse.cz> 写道:
>>
>>> [CC linux-mm]
>>>
>>> Hi,
>>>
>>> On Fri 13-04-12 18:00:10, gaoqiang wrote:
>>>>
>>>>
>>>> I put a single process into a cgroup and set memory.limit_in_bytes
>>>> to 100M,and memory.memsw.limit_in_bytes to 1G.
>>>>
>>>> howevery,the process was oom-killed before mem+swap hit 1G. I tried
>>>> many times,and it was killed randomly when memory+swap
>>>>
>>>> exceed 100M but less than 1G. what is the matter ?
>>>
>>> could you be more specific about your kernel version, workload and
>>> could
>>> you provide us with GROUP/memory.stat snapshots taken during your test?
>>>
>>> One reason for oom might be that you are hitting the hard limit (you
>>> cannot get over even if memsw limit says more) and you cannot swap out
>>> any pages (e.g. they are mlocked or under writeback).
>>>
>>
>> many thanks.
>>
>>
>> The system is a vmware virtual machine,running centos6.2 with kernel
>> 2.6.32-220.7.1.el6.x86_64.
>>
>> the attachments are memory.stat, the test program and the
>> /var/log/message of the oom.
>>
>> the workload is nearly 0,with searal sshd and bash program running.
>>
>> I just did the following command when testing:
>>
>> ./t
>> # this program will pause at the "getchar()" line and in another
>> terminal,run :
>>
>> cgclear
>> service cgconfig restart
>> mkdir /cgroup/memory/test
>> cd /cgroup/memory/test
>> echo 100m > memory.limit_in_bytes
>> echo 1G > memory.memsw.limit_in_bytes
>> echo 'pid' > tasks
>>
>> # then continue the t command
>>
>>
> Hi,
>
> I run your test under RHEL6.1 with 2.6.32-220.7.1.el6.x86_64 (an
> internal version but
> no changes in mm/memcg) in a real server and the process is killed with
> memsw reaching
> 1G. Does your vmware virtual machine have enough swap space?.. I've no
> idea whether
> the different behavior come from the physical/virtual environment.
>
>
> Thanks,
> Sha
>
>
--
使用 Opera 革命性的电子邮件客户程序: http://www.opera.com/mail/
[-- Attachment #2: messages --]
[-- Type: application/octet-stream, Size: 10753 bytes --]
Apr 12 19:43:25 c2 kernel: test invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
Apr 12 19:43:25 c2 kernel: test cpuset=/ mems_allowed=0-1
Apr 12 19:43:25 c2 kernel: Pid: 9867, comm: test Not tainted 2.6.32-220.7.1.el6.x86_64 #1
Apr 12 19:43:25 c2 kernel: Call Trace:
Apr 12 19:43:25 c2 kernel: [<ffffffff810c2c61>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Apr 12 19:43:25 c2 kernel: [<ffffffff811139e0>] ? dump_header+0x90/0x1b0
Apr 12 19:43:25 c2 kernel: [<ffffffff8120d7ac>] ? security_real_capable_noaudit+0x3c/0x70
Apr 12 19:43:25 c2 kernel: [<ffffffff81113e6a>] ? oom_kill_process+0x8a/0x2c0
Apr 12 19:43:25 c2 kernel: [<ffffffff81113da1>] ? select_bad_process+0xe1/0x120
Apr 12 19:43:25 c2 kernel: [<ffffffff81114602>] ? mem_cgroup_out_of_memory+0x92/0xb0
Apr 12 19:43:25 c2 kernel: [<ffffffff81169357>] ? mem_cgroup_handle_oom+0x147/0x170
Apr 12 19:43:25 c2 kernel: [<ffffffff81090a90>] ? autoremove_wake_function+0x0/0x40
Apr 12 19:43:25 c2 kernel: [<ffffffff8116a61b>] ? __mem_cgroup_try_charge+0x3bb/0x420
Apr 12 19:43:25 c2 kernel: [<ffffffff81123851>] ? __alloc_pages_nodemask+0x111/0x940
Apr 12 19:43:25 c2 kernel: [<ffffffff8116b917>] ? mem_cgroup_charge_common+0x87/0xd0
Apr 12 19:43:25 c2 kernel: [<ffffffff8116bae8>] ? mem_cgroup_newpage_charge+0x48/0x50
Apr 12 19:43:25 c2 kernel: [<ffffffff8113beca>] ? handle_pte_fault+0x79a/0xb50
Apr 12 19:43:25 c2 kernel: [<ffffffff810471c7>] ? pte_alloc_one+0x37/0x50
Apr 12 19:43:25 c2 kernel: [<ffffffff81171ad9>] ? do_huge_pmd_anonymous_page+0xb9/0x370
Apr 12 19:43:25 c2 kernel: [<ffffffff8113c464>] ? handle_mm_fault+0x1e4/0x2b0
Apr 12 19:43:25 c2 kernel: [<ffffffff81042b79>] ? __do_page_fault+0x139/0x480
Apr 12 19:43:25 c2 kernel: [<ffffffff8100988e>] ? __switch_to+0x26e/0x320
Apr 12 19:43:25 c2 kernel: [<ffffffff814ecb0e>] ? thread_return+0x4e/0x760
Apr 12 19:43:25 c2 kernel: [<ffffffff814f253e>] ? do_page_fault+0x3e/0xa0
Apr 12 19:43:25 c2 kernel: [<ffffffff814ef8f5>] ? page_fault+0x25/0x30
Apr 12 19:43:25 c2 kernel: Task in /9866 killed as a result of limit of /9866
Apr 12 19:43:25 c2 kernel: memory: usage 102400kB, limit 102400kB, failcnt 756
Apr 12 19:43:25 c2 kernel: memory+swap: usage 199656kB, limit 1024000kB, failcnt 0
Apr 12 19:43:25 c2 kernel: Mem-Info:
Apr 12 19:43:25 c2 kernel: Node 0 DMA per-cpu:
Apr 12 19:43:25 c2 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 4: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 5: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 6: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 7: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 8: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 9: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 10: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 11: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 12: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 13: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 14: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 15: hi: 0, btch: 1 usd: 0
Apr 12 19:43:25 c2 kernel: Node 0 DMA32 per-cpu:
Apr 12 19:43:25 c2 kernel: CPU 0: hi: 186, btch: 31 usd: 74
Apr 12 19:43:25 c2 kernel: CPU 1: hi: 186, btch: 31 usd: 34
Apr 12 19:43:25 c2 kernel: CPU 2: hi: 186, btch: 31 usd: 177
Apr 12 19:43:25 c2 kernel: CPU 3: hi: 186, btch: 31 usd: 164
Apr 12 19:43:25 c2 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 5: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 8: hi: 186, btch: 31 usd: 18
Apr 12 19:43:25 c2 kernel: CPU 9: hi: 186, btch: 31 usd: 33
Apr 12 19:43:25 c2 kernel: CPU 10: hi: 186, btch: 31 usd: 32
Apr 12 19:43:25 c2 kernel: CPU 11: hi: 186, btch: 31 usd: 167
Apr 12 19:43:25 c2 kernel: CPU 12: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 13: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 14: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 15: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: Node 0 Normal per-cpu:
Apr 12 19:43:25 c2 kernel: CPU 0: hi: 186, btch: 31 usd: 151
Apr 12 19:43:25 c2 kernel: CPU 1: hi: 186, btch: 31 usd: 163
Apr 12 19:43:25 c2 kernel: CPU 2: hi: 186, btch: 31 usd: 181
Apr 12 19:43:25 c2 kernel: CPU 3: hi: 186, btch: 31 usd: 180
Apr 12 19:43:25 c2 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 5: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 8: hi: 186, btch: 31 usd: 156
Apr 12 19:43:25 c2 kernel: CPU 9: hi: 186, btch: 31 usd: 114
Apr 12 19:43:25 c2 kernel: CPU 10: hi: 186, btch: 31 usd: 171
Apr 12 19:43:25 c2 kernel: CPU 11: hi: 186, btch: 31 usd: 160
Apr 12 19:43:25 c2 kernel: CPU 12: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 13: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 14: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 15: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: Node 1 Normal per-cpu:
Apr 12 19:43:25 c2 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 1: hi: 186, btch: 31 usd: 11
Apr 12 19:43:25 c2 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 4: hi: 186, btch: 31 usd: 53
Apr 12 19:43:25 c2 kernel: CPU 5: hi: 186, btch: 31 usd: 153
Apr 12 19:43:25 c2 kernel: CPU 6: hi: 186, btch: 31 usd: 40
Apr 12 19:43:25 c2 kernel: CPU 7: hi: 186, btch: 31 usd: 128
Apr 12 19:43:25 c2 kernel: CPU 8: hi: 186, btch: 31 usd: 134
Apr 12 19:43:25 c2 kernel: CPU 9: hi: 186, btch: 31 usd: 40
Apr 12 19:43:25 c2 kernel: CPU 10: hi: 186, btch: 31 usd: 5
Apr 12 19:43:25 c2 kernel: CPU 11: hi: 186, btch: 31 usd: 0
Apr 12 19:43:25 c2 kernel: CPU 12: hi: 186, btch: 31 usd: 162
Apr 12 19:43:25 c2 kernel: CPU 13: hi: 186, btch: 31 usd: 26
Apr 12 19:43:25 c2 kernel: CPU 14: hi: 186, btch: 31 usd: 60
Apr 12 19:43:25 c2 kernel: CPU 15: hi: 186, btch: 31 usd: 165
Apr 12 19:43:25 c2 kernel: active_anon:23933 inactive_anon:12772 isolated_anon:0
Apr 12 19:43:25 c2 kernel: active_file:2735681 inactive_file:2912499 isolated_file:0
Apr 12 19:43:25 c2 kernel: unevictable:0 dirty:5 writeback:12826 unstable:0
Apr 12 19:43:25 c2 kernel: free:1850423 slab_reclaimable:563455 slab_unreclaimable:22302
Apr 12 19:43:25 c2 kernel: mapped:3156 shmem:58 pagetables:2532 bounce:0
Apr 12 19:43:25 c2 kernel: Node 0 DMA free:15660kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15248kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 12 19:43:25 c2 kernel: lowmem_reserve[]: 0 2991 16121 16121
Apr 12 19:43:25 c2 kernel: Node 0 DMA32 free:1819872kB min:8344kB low:10428kB high:12516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3063392kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:841860kB slab_unreclaimable:11124kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 12 19:43:25 c2 kernel: lowmem_reserve[]: 0 0 13130 13130
Apr 12 19:43:25 c2 kernel: Node 0 Normal free:45632kB min:36632kB low:45788kB high:54948kB active_anon:19128kB inactive_anon:28kB active_file:6287228kB inactive_file:5978212kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13445120kB mlocked:0kB dirty:0kB writeback:0kB mapped:9252kB shmem:108kB slab_reclaimable:1065380kB slab_unreclaimable:31404kB kernel_stack:3504kB pagetables:3860kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 12 19:43:25 c2 kernel: lowmem_reserve[]: 0 0 0 0
Apr 12 19:43:25 c2 kernel: Node 1 Normal free:5520528kB min:45088kB low:56360kB high:67632kB active_anon:76604kB inactive_anon:51060kB active_file:4655496kB inactive_file:5671784kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16547840kB mlocked:0kB dirty:20kB writeback:51304kB mapped:3372kB shmem:124kB slab_reclaimable:346580kB slab_unreclaimable:46680kB kernel_stack:448kB pagetables:6268kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 12 19:43:25 c2 kernel: lowmem_reserve[]: 0 0 0 0
Apr 12 19:43:25 c2 kernel: Node 0 DMA: 3*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15660kB
Apr 12 19:43:25 c2 kernel: Node 0 DMA32: 66*4kB 109*8kB 47*16kB 24*32kB 28*64kB 17*128kB 7*256kB 8*512kB 9*1024kB 4*2048kB 437*4096kB = 1819872kB
Apr 12 19:43:25 c2 kernel: Node 0 Normal: 134*4kB 221*8kB 48*16kB 28*32kB 11*64kB 4*128kB 2*256kB 10*512kB 10*1024kB 2*2048kB 5*4096kB = 45632kB
Apr 12 19:43:25 c2 kernel: Node 1 Normal: 10*4kB 6*8kB 7*16kB 4*32kB 71*64kB 171*128kB 127*256kB 67*512kB 52*1024kB 2*2048kB 1311*4096kB = 5520776kB
Apr 12 19:43:25 c2 kernel: 5661062 total pagecache pages
Apr 12 19:43:25 c2 kernel: 12823 pages in swap cache
Apr 12 19:43:25 c2 kernel: Swap cache stats: add 37243, delete 24420, find 57/66
Apr 12 19:43:25 c2 kernel: Free swap = 34946028kB
Apr 12 19:43:25 c2 kernel: Total swap = 35094520kB
Apr 12 19:43:25 c2 kernel: 8388607 pages RAM
Apr 12 19:43:25 c2 kernel: 171092 pages reserved
Apr 12 19:43:25 c2 kernel: 5343598 pages shared
Apr 12 19:43:25 c2 kernel: 1039448 pages non-shared
Apr 12 19:43:25 c2 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 12 19:43:25 c2 kernel: [ 9866] 0 9866 21650 455 13 0 0 cglimit
Apr 12 19:43:25 c2 kernel: [ 9867] 0 9867 58345 12846 4 0 0 test
Apr 12 19:43:25 c2 kernel: Memory cgroup out of memory: Kill process 9867 (test) score 1000 or sacrifice child
Apr 12 19:43:25 c2 kernel: Killed process 9867, UID 0, (test) total-vm:233380kB, anon-rss:51004kB, file-rss:380kB
prev parent reply other threads:[~2012-04-17 8:20 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <op.wco7ekvhn27o5l@gaoqiang-d1.corp.qihoo.net>
2012-04-13 14:49 ` Michal Hocko
2012-04-16 3:43 ` gaoqiang
2012-04-16 14:26 ` Michal Hocko
2012-04-17 3:25 ` Sha Zhengju
2012-04-17 8:18 ` gaoqiang [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=op.wcwhcri3n27o5l@gaoqiang-d1.corp.qihoo.net \
--to=gaoqiangscut@gmail.com \
--cc=cgroups@vger.kernel.org \
--cc=handai.szj@gmail.com \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox