From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32C30C4332F for ; Tue, 13 Dec 2022 13:30:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAC988E0003; Tue, 13 Dec 2022 08:30:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A35248E0002; Tue, 13 Dec 2022 08:30:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 888AD8E0003; Tue, 13 Dec 2022 08:30:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 73CA18E0002 for ; Tue, 13 Dec 2022 08:30:51 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 38E3B40AD7 for ; Tue, 13 Dec 2022 13:30:51 +0000 (UTC) X-FDA: 80237368302.05.CFB058B Received: from mail-ed1-f66.google.com (mail-ed1-f66.google.com [209.85.208.66]) by imf19.hostedemail.com (Postfix) with ESMTP id 0F43C1A0020 for ; Tue, 13 Dec 2022 13:30:48 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=zXfkNNv5; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf19.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.208.66 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670938249; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GkAga1fPXNCvgWWr0y17vOcwxqzOIKIhEW49YSLUBLs=; b=uF/tMXsx67mZBWE31isZuOD7M9Ztxt+CjlZBQoOXZBkOAlKGwDtvgIJL+LTQa0aehvfDsl PIMytnyOkosfkstAuR4UkCmhpUHPWxmppuHRIAv1QjoV8VbhW4LgwuxOvzO93RfjGtKn2p EzoyHAUcPJijiq6f47WaxNMd8sYp6IQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=zXfkNNv5; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf19.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.208.66 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670938249; a=rsa-sha256; cv=none; b=k3a+NIkzLVainpY9w0ouZBNyF/vi35Zj58vffNgu1I9XwT/2LID0qnTQgT6pKQNRcA6ukQ TzF4nOQOeHlRAVT6itrNzkWOe51HSKVTbIRlQArxMlk4TcZnDb9/MX68K+CdZCJ0jyvl+O cQWhGKgstOH/nkRd6lyB6g4QY+dg6Yg= Received: by mail-ed1-f66.google.com with SMTP id e13so17712721edj.7 for ; Tue, 13 Dec 2022 05:30:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=GkAga1fPXNCvgWWr0y17vOcwxqzOIKIhEW49YSLUBLs=; b=zXfkNNv5mXOMPEanvux+4bWlBQnFuzxE2MzUd22vOUfuSUzuyhRUOfDzB0ggA71T8F U4BoSvDNvouPUsFnxzBhZ9PkJKNOOi+r+fWw9vIFexrfG0KwcjqamyWJH78bxvSa+wEK sviWJZdff4IfSH7PcbdsfV9IggXAnSU97epnS/d0NovCmFSRag3LxiwLBWEbw0wia48V YbDPBVUWD+T8qS9AWX0q7GxruKFPjvBYZuFfVuhZJcW2vVgQJ/wUPa1YAkGfHLEO3TNv ut45RL2Pkoauar+k5MQKvHAfX+4LQXjfZ+Tr4d/sXdDmzrGpjYfmw5ayEfmXxR2ffeRd 2uuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=GkAga1fPXNCvgWWr0y17vOcwxqzOIKIhEW49YSLUBLs=; b=ZuUKHbxqjZv8D/H+1OKt4gr5lPkrB5NzzbxaX7iyJtZT26ejt2a4E3BVGiOl15JWIZ vit7c6oBgmdrZNZO5IPSF/oPRUI2Gp8gwzCOYsJcYV7RXUy4fjwfQATmwlfB9h/gzrXM ChIyyCBnLqIEtBGRoSCg/5WFEz7fFwLnSnwDZslf7lJziahe1UMftAFCFaXaDcw8RhPE 7URI0RoagCv4UDYFm30LHcYQE5LgOPThAdqL1K2sYyh1P7zvrs+v6WNR0sZGCFMp0I+d 5nVobk+k1fjDtve5X3xULshWtdG7lmRitAU5dT2CQlIEELWI6DZAEnMwlLs0+3dM6M+p y6ZQ== X-Gm-Message-State: ANoB5pknS/lp9HsA8nrmyTgoSdfEstI9hFeQ2ftBHKylmDE2WgK24+RL paTafVOaKcsgJ8aIb5bAHQB8xw== X-Google-Smtp-Source: AA0mqf55nZH+ZFdawcdqg2GbkWyleEiww6bqn+BU/p1ARK9riCKQWs9Ju9dLnhFZ+gdPQNXQCWCLww== X-Received: by 2002:a05:6402:c08:b0:46d:83ea:44e8 with SMTP id co8-20020a0564020c0800b0046d83ea44e8mr18308556edb.21.1670938247443; Tue, 13 Dec 2022 05:30:47 -0800 (PST) Received: from localhost (ip-046-005-139-011.um12.pools.vodafone-ip.de. [46.5.139.11]) by smtp.gmail.com with ESMTPSA id d7-20020aa7ce07000000b0046951b43e84sm4945096edv.55.2022.12.13.05.30.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Dec 2022 05:30:47 -0800 (PST) Date: Tue, 13 Dec 2022 14:30:40 +0100 From: Johannes Weiner To: "Huang, Ying" Cc: Mina Almasry , Michal Hocko , Tejun Heo , Zefan Li , Jonathan Corbet , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Yang Shi , Yosry Ahmed , weixugc@google.com, fvdl@google.com, bagasdotme@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v3] mm: Add nodes= arg to memory.reclaim Message-ID: References: <20221202223533.1785418-1-almasrymina@google.com> <87k02volwe.fsf@yhuang6-desk2.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87k02volwe.fsf@yhuang6-desk2.ccr.corp.intel.com> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0F43C1A0020 X-Stat-Signature: 3ngforn786cq79k3ffpymp1o8tu7j9q3 X-HE-Tag: 1670938248-204442 X-HE-Meta: U2FsdGVkX1/LMczjlmkyECIGs9LR2QlLd55H8nSxiovT4VLC5ngobeQMPJsaYLcvZwrQe/y0+G9UG7gwwzKbBICjR41EERkqSpcF/d4pq4AeZiPSctnoGPNgZ2i0+Tsabji2B7h551sE3cr5mFHFQOUzK6oK60jbZsavZAWN2rQYudtRyFWU5IL7rk40xr+pZXZjpL09Mh+fr76e37QRqeyhtWOsszrbGQpcqFWOUn6rCrtbC8U8eC0CyY2hb/VUs0ThLLhI6t4A7q8w8jlducNnlD09uOuiE5w9DEgEDXjSi5PSIZTg9UblvydW8hVb+OFq3fC4nUhw/lLGBPInqYenGqMIl4M+Qu/fRtSdGHWFMJMXT9X8i5L00H3jEJBpiNcbxbJMAeWQtOTFxwoAKgAcjuZOjhFhPhCPXv4EkTSgnWwIAqHbOkDQFcyznibEyAwmu1AR37wBZo5sDAZnikGKzhF6Y8HQr1VXldSxpMevaspD77NOQ8gnbTJCznNAq8+cUHs47V08Yys3Qqdg7P2SEY9ow5HbQZAPfYhrqf6noIXlVLZLlsJZic8hyO4gzCCnJ9SkpSJHrlxhvdy88pY7/Gt9Bl40h1+P1WWKG6nvFz6puFXQaw1d1LJpgDB4frPLNXmtDq+F9/UzojzEM006iiTrrK14WuP30nU2Yxr/5pAHs2cCeZNeGFXeSByMpM1+PpRBftL8g9kjTM0DE3GW6i2xV7aLqWOKbWnxaj1hXD3ixLtGqmnF/XzHmhYa7Hpq+SMPuBjtejE+b9S5gjR7+6l7SA6Gh+b+X+8BJHXwzSf0TgFETvxQ9w68sJp2NtPEC0+Uaxok/6KEcxhWpzgfQZcokTMfjNI5qbZxtVplD4eZwsGSCHzeKZw5r1hoq9ei3SuWpfmsKEOvPlACTB0KHLMObIoM7Leu83SmykHyB4Qymji4Bxk36WB1r9K9uQsimE8vnvHVw5p+pi1 8oXsS5/b e8f9j5yTekAPYa0MoZMMXUHU7IYebPZiHYEZmOfS3WuaIAoN78AEmUhMaCB5/UEjzHP8c62H2mS17OZdwgaf+yByfHKZg0VQ4MrK5llTgzdAkyh0KND8fUg3PviWzBs7mAFpphEXfAUOvPJJAeHBjVsEb6vAz4u2BYkJdJVCT/QBf3IbSDvplnMnc2J73Mzv7gzHVhxcvgOcAA7+AWlfmBrWSGA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Dec 13, 2022 at 02:30:57PM +0800, Huang, Ying wrote: > Mina Almasry writes: > > > On Mon, Dec 12, 2022 at 12:55 AM Michal Hocko wrote: > >> > >> On Fri 02-12-22 14:35:31, Mina Almasry wrote: > >> > The nodes= arg instructs the kernel to only scan the given nodes for > >> > proactive reclaim. For example use cases, consider a 2 tier memory system: > >> > > >> > nodes 0,1 -> top tier > >> > nodes 2,3 -> second tier > >> > > >> > $ echo "1m nodes=0" > memory.reclaim > >> > > >> > This instructs the kernel to attempt to reclaim 1m memory from node 0. > >> > Since node 0 is a top tier node, demotion will be attempted first. This > >> > is useful to direct proactive reclaim to specific nodes that are under > >> > pressure. > >> > > >> > $ echo "1m nodes=2,3" > memory.reclaim > >> > > >> > This instructs the kernel to attempt to reclaim 1m memory in the second tier, > >> > since this tier of memory has no demotion targets the memory will be > >> > reclaimed. > >> > > >> > $ echo "1m nodes=0,1" > memory.reclaim > >> > > >> > Instructs the kernel to reclaim memory from the top tier nodes, which can > >> > be desirable according to the userspace policy if there is pressure on > >> > the top tiers. Since these nodes have demotion targets, the kernel will > >> > attempt demotion first. > >> > > >> > Since commit 3f1509c57b1b ("Revert "mm/vmscan: never demote for memcg > >> > reclaim""), the proactive reclaim interface memory.reclaim does both > >> > reclaim and demotion. Reclaim and demotion incur different latency costs > >> > to the jobs in the cgroup. Demoted memory would still be addressable > >> > by the userspace at a higher latency, but reclaimed memory would need to > >> > incur a pagefault. > >> > > >> > The 'nodes' arg is useful to allow the userspace to control demotion > >> > and reclaim independently according to its policy: if the memory.reclaim > >> > is called on a node with demotion targets, it will attempt demotion first; > >> > if it is called on a node without demotion targets, it will only attempt > >> > reclaim. > >> > > >> > Acked-by: Michal Hocko > >> > Signed-off-by: Mina Almasry > >> > >> After discussion in [1] I have realized that I haven't really thought > >> through all the consequences of this patch and therefore I am retracting > >> my ack here. I am not nacking the patch at this statge but I also think > >> this shouldn't be merged now and we should really consider all the > >> consequences. > >> > >> Let me summarize my main concerns here as well. The proposed > >> implementation doesn't apply the provided nodemask to the whole reclaim > >> process. This means that demotion can happen outside of the mask so the > >> the user request cannot really control demotion targets and that limits > >> the interface should there be any need for a finer grained control in > >> the future (see an example in [2]). > >> Another problem is that this can limit future reclaim extensions because > >> of existing assumptions of the interface [3] - specify only top-tier > >> node to force the aging without actually reclaiming any charges and > >> (ab)use the interface only for aging on multi-tier system. A change to > >> the reclaim to not demote in some cases could break this usecase. > >> > > > > I think this is correct. My use case is to request from the kernel to > > do demotion without reclaim in the cgroup, and the reason for that is > > stated in the commit message: > > > > "Reclaim and demotion incur different latency costs to the jobs in the > > cgroup. Demoted memory would still be addressable by the userspace at > > a higher latency, but reclaimed memory would need to incur a > > pagefault." > > > > For jobs of some latency tiers, we would like to trigger proactive > > demotion (which incurs relatively low latency on the job), but not > > trigger proactive reclaim (which incurs a pagefault). I initially had > > proposed a separate interface for this, but Johannes directed me to > > this interface instead in [1]. In the same email Johannes also tells > > me that meta's reclaim stack relies on memory.reclaim triggering > > demotion, so it seems that I'm not the first to take a dependency on > > this. Additionally in [2] Johannes also says it would be great if in > > the long term reclaim policy and demotion policy do not diverge. > > > > [1] https://lore.kernel.org/linux-mm/Y35fw2JSAeAddONg@cmpxchg.org/ > > [2] https://lore.kernel.org/linux-mm/Y36fIGFCFKiocAd6@cmpxchg.org/ > > After these discussion, I think the solution maybe use different > interfaces for "proactive demote" and "proactive reclaim". That is, > reconsider "memory.demote". In this way, we will always uncharge the > cgroup for "memory.reclaim". This avoid the possible confusion there. > And, because demotion is considered aging, we don't need to disable > demotion for "memory.reclaim", just don't count it. Hm, so in summary: 1) memory.reclaim would demote and reclaim like today, but it would change to only count reclaimed pages against the goal. 2) memory.demote would only demote. a) What if the demotion targets are full? Would it reclaim or fail? 3) Would memory.reclaim and memory.demote still need nodemasks? Would they return -EINVAL if a) memory.reclaim gets passed only toptier nodes or b) memory.demote gets passed any lasttier nodes?