From: Nishanth Aravamudan <nacc@us.ibm.com>
To: Christoph Lameter <clameter@sgi.com>
Cc: lee.schermerhorn@hp.com, wli@holomorphy.com, melgor@ie.ibm.com,
akpm@linux-foundation.org, linux-mm@kvack.org, agl@us.ibm.com,
pj@sgi.com
Subject: Re: [RFC][PATCH 4/5] hugetlb: fix cpuset-constrained pool resizing
Date: Tue, 7 Aug 2007 18:50:42 -0700 [thread overview]
Message-ID: <20070808015042.GF15714@us.ibm.com> (raw)
In-Reply-To: <Pine.LNX.4.64.0708061101470.24256@schroedinger.engr.sgi.com>
On 06.08.2007 [11:04:48 -0700], Christoph Lameter wrote:
> On Mon, 6 Aug 2007, Nishanth Aravamudan wrote:
>
> > hugetlb: fix cpuset-constrained pool resizing
> >
> > With the previous 3 patches in this series applied, if a process is in a
> > constrained cpuset, and tries to grow the hugetlb pool, hugepages may be
> > allocated on nodes outside of the process' cpuset. More concretely,
> > growing the pool via
> >
> > echo some_value > /proc/sys/vm/nr_hugepages
> >
> > interleaves across all nodes with memory such that hugepage allocations
> > occur on nodes outside the cpuset. Similarly, this process is able to
> > change the values in values in
> > /sys/devices/system/node/nodeX/nr_hugepages, even when X is not in the
> > cpuset. This directly violates the isolation that cpusets is supposed to
> > guarantee.
>
> No it does not. Cpusets do not affect the administrative rights of users.
For reference here (as I just ran my simple script against
2.6.23-rc1-mm2, 2.6.23-rc1-mm2 + your patches, 2.6.23-rc1-mm2 + your
patches + each of my patches in turn), this is completely untrue with
-mm2 and your patches. I was actually trying to restore this behavior
with this patch. I realize I didn't mention this earlier... On a 4-node
x86_64:
2.6.23-rc1-mm2:
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 100
Node 0 HugePages_Free: 0
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
2.6.23-rc1-mm2 + your patches:
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 100
Node 0 HugePages_Free: 0
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
After my patch 1/2 (try harder) from this morning:
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 25
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 75
Node 0 HugePages_Free: 25
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 0
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 0
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 25
Node 2 HugePages_Free: 100
Node 1 HugePages_Free: 50
Node 0 HugePages_Free: 25
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 25
Node 2 HugePages_Free: 100
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 25
Done. 150 free
After patch 2/2 (memoryless nodes) from this morning (the results are
actually the same as the above, just that the values are shifted around
the nodes a bit):
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 100
Node 2 HugePages_Free: 50
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 100
Node 2 HugePages_Free: 50
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 25
Done. 175 free
Finally, after my hugetlb interleave dequeue patch is applied:
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 50
Node 2 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 0 HugePages_Free: 50
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 25
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 50
Node 2 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 0 HugePages_Free: 50
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 0
Node 2 HugePages_Free: 0
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 0 free
So, it would appear that, in your opinion, this set of patches
constitutes a pseudo-bug-fix? Without the last patch, it seems, cpusets
are able to constrain what nodes a root process can remove hugepages
from.
Thanks,
Nish
--
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2007-08-08 1:50 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-08-06 16:32 [RFC][PATCH 0/5] hugetlb NUMA improvements Nishanth Aravamudan
2007-08-06 16:37 ` [RFC][PATCH 1/5] Fix hugetlb pool allocation with empty nodes V9 Nishanth Aravamudan
2007-08-06 16:38 ` [RFC][PATCH 2/5] hugetlb: numafy several functions Nishanth Aravamudan
2007-08-06 16:40 ` [RFC][PATCH 3/5] hugetlb: add per-node nr_hugepages sysfs attribute Nishanth Aravamudan
2007-08-06 16:44 ` [RFC][PATCH 4/5] hugetlb: fix cpuset-constrained pool resizing Nishanth Aravamudan
2007-08-06 16:45 ` Nishanth Aravamudan
2007-08-06 16:48 ` [RFC][PATCH 5/5] hugetlb: interleave dequeueing of huge pages Nishanth Aravamudan
2007-08-06 18:04 ` [RFC][PATCH 4/5] hugetlb: fix cpuset-constrained pool resizing Christoph Lameter
2007-08-06 18:26 ` Nishanth Aravamudan
2007-08-06 18:41 ` Christoph Lameter
2007-08-07 0:03 ` Nishanth Aravamudan
2007-08-06 19:37 ` Lee Schermerhorn
2007-08-08 1:50 ` Nishanth Aravamudan [this message]
2007-08-08 13:26 ` Lee Schermerhorn
2007-08-06 17:59 ` [RFC][PATCH 2/5] hugetlb: numafy several functions Christoph Lameter
2007-08-06 18:15 ` Nishanth Aravamudan
2007-08-07 0:34 ` Nishanth Aravamudan
2007-08-06 18:00 ` [RFC][PATCH 1/5] Fix hugetlb pool allocation with empty nodes V9 Christoph Lameter
2007-08-06 18:19 ` Nishanth Aravamudan
2007-08-06 18:37 ` Christoph Lameter
2007-08-06 19:52 ` Lee Schermerhorn
2007-08-06 20:15 ` Christoph Lameter
2007-08-07 0:04 ` Nishanth Aravamudan
2007-08-06 16:39 ` [RFC][PATCH 0/5] hugetlb NUMA improvements Nishanth Aravamudan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20070808015042.GF15714@us.ibm.com \
--to=nacc@us.ibm.com \
--cc=agl@us.ibm.com \
--cc=akpm@linux-foundation.org \
--cc=clameter@sgi.com \
--cc=lee.schermerhorn@hp.com \
--cc=linux-mm@kvack.org \
--cc=melgor@ie.ibm.com \
--cc=pj@sgi.com \
--cc=wli@holomorphy.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox