From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1B66C38145 for ; Wed, 7 Sep 2022 13:50:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E6476B0072; Wed, 7 Sep 2022 09:50:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 295DD8D0003; Wed, 7 Sep 2022 09:50:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15DF88D0002; Wed, 7 Sep 2022 09:50:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 095266B0072 for ; Wed, 7 Sep 2022 09:50:40 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D4C1FA113D for ; Wed, 7 Sep 2022 13:50:39 +0000 (UTC) X-FDA: 79885424598.18.16B7206 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf05.hostedemail.com (Postfix) with ESMTP id 2114910008C for ; Wed, 7 Sep 2022 13:50:35 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id z9-20020a17090a468900b001ffff693b27so13449308pjf.2 for ; Wed, 07 Sep 2022 06:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to:subject :user-agent:mime-version:date:message-id:from:to:cc:subject:date; bh=MgiF2QhoVH7izaAC6v9HdSeip3/dnqOw2S5EnYEL7zk=; b=tyBysNtakVLwLosd6NM/vK1CqUudzQqZPlC72cw1rfa2hg2wgZvjK0mTMetrBlukH7 tZb+67+N/zPM5Tmb+bOM1Sykm7wQgEVgj8BUBD8vCGwxoSYiRU/hUjveit3Ud+/2bmBC ZR7Ssg78La9RTA7wxRojAleaPa7l0715zdhc2pbBLmjwJTFUPto3Fv10Vvy6GhelSdFG A+8ZMtgwXszLUMihVXzN7kstNCgHJEzSn/Aw+XPw44wzu6MDLjAWgNOu8Ej6gsuX4eOm RTYQun6Tw78tSBtvWIMsySt1k72KvjHw8OpB3O1h7WMhOXb/pDiZ/L+kDny2Jt3KubrT H/GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to:subject :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date; bh=MgiF2QhoVH7izaAC6v9HdSeip3/dnqOw2S5EnYEL7zk=; b=6n+RURuCSP90yo6i6TJ0aCvzDxg5Ljs3E6Dy3CV4/ixfG9DQpmZJ4wDBYkjzfYoakx 9rKrgjgDN5Vh2Qb8bPwU0zpH+llGOWrpHZWyrhH2HQMHwM/ctYDQ/AmcQlJG5AzlreSm 4jh8CLzZqsGqZ6Ji/f/JswTZJ/UN08ec7B9KfBVGBrSg/wmvfHMPhcMn5cWZwjm5L8+V ruV8229x0E/TrEOPM6yDuVy+6c23eIsGeeUm7SYXpccoKtFv76EV0sXiboT9ygp0nzOT NiigDJakE6+JGk5nWelj4dcZnuO3kmHyJAJjPB/PLjSFM9qaoif5yLds1tC5PukizBKW vjLg== X-Gm-Message-State: ACgBeo08MN5NF3IbBa/YTkUTUdIlIKI6foOJahCXz4LSBupDtN2+x/yH i2p4JJWYUwD2r5u5rxvjM6OMfg== X-Google-Smtp-Source: AA6agR5koSkOslLJYDBSVR9RGjjb1QUMj/VxBP5Lmxqdsv+78ymCjMJHq0Ni7SLIHIhMMViosj+3ig== X-Received: by 2002:a17:90a:c1:b0:1f4:f757:6b48 with SMTP id v1-20020a17090a00c100b001f4f7576b48mr30371570pjd.56.1662558633815; Wed, 07 Sep 2022 06:50:33 -0700 (PDT) Received: from [10.255.85.171] ([139.177.225.235]) by smtp.gmail.com with ESMTPSA id a7-20020aa78e87000000b0053bb934eaa9sm8324011pfr.201.2022.09.07.06.50.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 07 Sep 2022 06:50:33 -0700 (PDT) Message-ID: <93d76370-6c43-5560-9a5f-f76a8cc979e0@bytedance.com> Date: Wed, 7 Sep 2022 21:50:24 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.13.0 Subject: Re: [External] Re: [PATCH] cgroup/cpuset: Add a new isolated mems.policy type. To: Michal Hocko Cc: hannes@cmpxchg.org, roman.gushchin@linux.dev, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, lizefan.x@bytedance.com, wuyun.abel@bytedance.com References: <20220904040241.1708-1-hezhongkun.hzk@bytedance.com> <0e5f380b-9201-0f56-9144-ce8449491fc8@bytedance.com> From: Zhongkun He In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662558639; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MgiF2QhoVH7izaAC6v9HdSeip3/dnqOw2S5EnYEL7zk=; b=ZJie1DMJywPSELzLAWRCth5nljrUiS2wPCJaBLB/5SksfF54ULXME2BM9lQZZnyrDaANfe U+FH4+u59vGzPLllRxQQ8sAnGK+4ptWHk5rUtD7Kjx4Q4EMhsrDWllgXlvhgW9nu662MIQ Z7kZqinnsE2hFxmRapanxwzr6KWOGuQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=tyBysNta; spf=pass (imf05.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662558639; a=rsa-sha256; cv=none; b=0AS59+VO/1oLbvD6NgQgqfS1ZjttswHHcPt8grnNvI0WiPQqJPWelGxBjPRK8OCEGTFt25 S+EzWIzPelHYOs0Z9A6lJqnKiLc4U2BIFKUSd0nonqbavLu+2pSWIBzv22/fJLrTq1xOju 5sIiLa/Qbt6eVrO1ZPdLZeVJGFPCcnM= X-Stat-Signature: s9raxrjgzpxtjip5a5kj764qpfywfu1h X-Rspamd-Queue-Id: 2114910008C X-Rspam-User: Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=tyBysNta; spf=pass (imf05.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam09 X-HE-Tag: 1662558635-242399 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: >> Hi Michal, thanks for your reply. >> >>>> Say parent has a stronger requirement (say bind) than a child(prefer)? >> >> Yes, combine all these together. > > What is the semantic of the resulting policy? > >> The parent's task will use 'bind', child's >> use 'prefer'.This is the current implementation, and we can discuss and >> modify it together if there are other suggestions. >> >> 1:Existing shortcomings >> >> In our use case, the application and the control plane are two separate >> systems. When the application is created, it doesn't know how to use memory, >> and it doesn't care. The control plane will decide the memory usage policy >> based on different reasons (the attributes of the application itself, the >> priority, the remaining resources of the system). Currently, numactl is used >> to set it at program startup, and the child process will inherit the >> mempolicy. > > Yes this is common practice I have seen so far. > >> But we can't dynamically adjust the memory policy, except >> restart, the memory policy will not change. > > Do you really need to change the policy itself or only the effective > nodemask? I mean what is your usecase to go from say mbind to preferred > policy? Do you need any other policy than bind and preferred? > >> 2:Our goals >> >> For the above reasons, we want to create a mempolicy at the cgroup level. >> Usually processes under a cgroup have the same priority and attributes, and >> we can dynamically adjust the memory allocation strategy according to the >> remaining resources of the system. For example, a low-priority cgroup uses >> the 'bind:2-3' policy, and a high-priority cgroup uses bind:0-1. When >> resources are insufficient, it will be changed to bind:3, bind:0-2 by >> control plane, etc.Further more, more mempolicy can be extended, such as >> allocating memory according to node weight, etc. > > Yes, I do understand that you want to change the node affinity and that > is already possible with cpuset cgroup. The existing constrain is that > the policy is hardcoded mbind IIRC. So you cannot really implement a dynamic > preferred policy which would make some sense to me. The question is how > to implement that with a sensible semantic. It is hard to partition the > system into several cgroups if subset allows to spill over to others. > Say something like the following > root (nodes=0-3) > / \ > A (0, 1) B (2, 3) > > if both are MBIND then this makes sense because they are kinda isolated > (at least for user allocations) but if B is PREFERRED and therefore > allowed to use nodes 0 and 1 then it can deplete the memory from A and > therefore isolation doesn't work at all. > > I can imagine that the all cgroups would use PREFERRED policy and then > nobody can expect anything and the configuration is mostly best effort. > But it feels like this is an abuse of the cgroup interface and a proper > syscall interface is likely due. Would it make more sense to add > pidfd_set_mempolicy and allow sufficiently privileged process to > manipulate default memory policy of a remote process? Hi Michal, thanks for your reply. > Do you really need to change the policy itself or only the effective > nodemask? Do you need any other policy than bind and preferred? Yes, we need to change the policy, not only his nodemask. we really want policy is interleave, and extend it to weight-interleave. Say something like the following node weight interleave: 0-3 1:1:1:1 default one by one weight-interleave: 0-3 1:2:4:6 alloc pages by weight (User set weight.) In the actual usecase, the remaining resources of each node are different, and the use of interleave cannot maximize the use of resources. Back to the previous question. >The question is how to implement that with a sensible semantic. Thanks for your analysis and suggestions.It is really difficult to add policy directly to cgroup for the hierarchical enforcement. It would be a good idea to add pidfd_set_mempolicy. Also, there is a new idea. We can try to separate the elements of mempolicy and use them independently. Mempolicy has two meanings: nodes:which nodes to use(nodes,0-3), we can use cpuset's effective_mems directly. mode:how to use them(bind,prefer,etc). change the mode to a cpuset->flags,such as CS_INTERLEAVE。 task_struct->mems_allowed is equal to cpuset->effective_mems,which is hierarchical enforcement。CS_INTERLEAVE can also be updated into tasks， just like other flags(CS_SPREAD_PAGE). When a process needs to allocate memory, it can find the appropriate node to allocate pages according to the flag and mems_allowed. thanks.