From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42104C00145 for ; Thu, 15 Dec 2022 05:51:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7DAE98E0003; Thu, 15 Dec 2022 00:51:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78B858E0002; Thu, 15 Dec 2022 00:51:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 654188E0003; Thu, 15 Dec 2022 00:51:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 51D788E0002 for ; Thu, 15 Dec 2022 00:51:21 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 25BB9140208 for ; Thu, 15 Dec 2022 05:51:21 +0000 (UTC) X-FDA: 80243467962.04.818B699 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf23.hostedemail.com (Postfix) with ESMTP id 7D0D3140004 for ; Thu, 15 Dec 2022 05:51:17 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=cNKB4PGD; spf=pass (imf23.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671083478; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w5ILZqQ09vmS5CC/QZ9eGGmUsyPymwqAt4wV4mrCWGY=; b=P8t95YMeDq8goUwCYT3T2bwr4a6ySlrtOgOiPIvv+l6R/5oF801vnAhrLVBUOUqJgTDYz5 xCjKzU9SSaM+1Wtj/wIrMmwIcK0YFB/NklsMYhqBPWHE7ynYagOiRXP2r0B4id3V2yFQbN N360p9YTN/Vh7ggvleLrWm7QTMfSd1s= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=cNKB4PGD; spf=pass (imf23.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671083478; a=rsa-sha256; cv=none; b=xZuMT5M1X3w/3y2hxHVmUzIMlulODKenSg3eUHMQwfz6EaJ1H4WtrUkiX7gNknHoJpyM4j SnlhlbqjQRYmyyR5aqFqnyYnrJAyXcjVtSiLPMdq+HmJtLlCZZcF1eaQ82usztL7l8GUIq TGQT9FcJMHSaSoWdmFy6fAN3PGMrwaU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1671083477; x=1702619477; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=Vy1nnsBUMyfJdWB0HL7AxX9ZAnGiu08F+Oqh6ALowk0=; b=cNKB4PGDHiqoUNgK/++4myv/eTBMbhNDBhId+76p/a9rcwnjTuEeqv2O hs3YgCHVGqKOtzHvV/Co6aMfeoOQ+WDPfwKNsxbuLjV39Ms1cUQSAywiC kdIGGkG81i2T/WTQsU5/BUoqXuM2JB/2OzjQqmxuoqUKCwt1VYg9eoiU6 qx9q/N9lF9U38379dxvoVHMLkrxzGe4Xx9TJJw3cRBr1U58jjbcdm8n5H ioFRyfzHRLdH+AO/SqV3Ccnvx/ELa9Iitihmpi+wwy4z7WIPthrQfbAoc 2Q+HV01BcE+HKxxdIWSPtEMiTrkH/AKGKyuerjBnNHqw/nmWMGrHuHeNE g==; X-IronPort-AV: E=McAfee;i="6500,9779,10561"; a="320448566" X-IronPort-AV: E=Sophos;i="5.96,246,1665471600"; d="scan'208";a="320448566" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Dec 2022 21:51:15 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10561"; a="978080190" X-IronPort-AV: E=Sophos;i="5.96,246,1665471600"; d="scan'208";a="978080190" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Dec 2022 21:51:10 -0800 From: "Huang, Ying" To: Michal Hocko Cc: Mina Almasry , Johannes Weiner , Tejun Heo , Zefan Li , Jonathan Corbet , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Yang Shi , Yosry Ahmed , weixugc@google.com, fvdl@google.com, bagasdotme@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v3] mm: Add nodes= arg to memory.reclaim References: <20221202223533.1785418-1-almasrymina@google.com> <87k02volwe.fsf@yhuang6-desk2.ccr.corp.intel.com> Date: Thu, 15 Dec 2022 13:50:14 +0800 In-Reply-To: (Michal Hocko's message of "Wed, 14 Dec 2022 11:23:32 +0100") Message-ID: <87mt7pdxm1.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7D0D3140004 X-Stat-Signature: o5tndweaequnm8iymeqxpfsfqhujbqec X-Rspam-User: X-HE-Tag: 1671083477-490617 X-HE-Meta: U2FsdGVkX1+EITjuPeTVE+536ywVkxRMG3w8Y2V2BibIGInvdBLQ4yDAQT+s4vjEqEXFP8qQo9p5/QsZETWqVExghIwngO2OIuwlLav8NXfh6MTLtPm4Uje1yl3Ff+GaBVmfrgeBKkXNzICfudSrPuNT7F3yC1Xi90I7pnc6Kf8C6g0Pfn4iAe9rkaG0b4Y/bqrSUJgy/3Br2+Bya2rR0wVEWWzcG+kxYyg77DYaZaXnYVbwBsp4a3agneqEufHPVB7aUA15J79Uj1tsl2tVSvIaTfFmMAxZ0P52U4Rz5qjObcJCruQVbyw4jVPnYZKh9zY7GdTc3UPO5S4po65CA8eH1or1MxEzj+Fsn+qit+eJ3LqdruaQ5UWjHYKpaxPz5w7g8/EjJlnnBeMBAvFgPqO9CHH6JD1kSM63dIs/6WhJqjdBwXo9Ml1r7AdWEUx36crMyX/vBWgUgyGTyKadIs54GCfGRRehO4ES8K6KZioEkLOkXa7IZUqv0WwEJJfV0nRfXAzjZC+Kwv/9qNs7xiJvw9dQk5EIQ/wq8BXrsPPZxoteiqx9Bpcrft7KjTBl0CFgDlSSKVm1W3IQ/vvB8URrEBts9mwFWrL3T92n1SgkFv4gp3g1mQB1Q11KcDeh5WWag5qQno1eV+GJKvGuLqD9AAKTCtFx+MZj5LqHcf9Cy2kvNsedisqZ/JOZ4vw/H6FSm5XCMEkbTuPzQ7UoPK//qKYpx+u7FgcPblBABz4fETv2Ro2Vt7szCYVfshzywlBGIfxLBi73bovk5NXWDvCF5zvR37xiXjeBZc6GIHEcZ3zaPgKZVtOEscg+1XLWh2btU3TFjFczM94w/INGumwit7kVNBi4uyqPaOs6GpWUkvPmEGQZSQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Michal Hocko writes: > On Tue 13-12-22 11:29:45, Mina Almasry wrote: >> On Tue, Dec 13, 2022 at 6:03 AM Michal Hocko wrote: >> > >> > On Tue 13-12-22 14:30:40, Johannes Weiner wrote: >> > > On Tue, Dec 13, 2022 at 02:30:57PM +0800, Huang, Ying wrote: >> > [...] >> > > > After these discussion, I think the solution maybe use different >> > > > interfaces for "proactive demote" and "proactive reclaim". That is, >> > > > reconsider "memory.demote". In this way, we will always uncharge the >> > > > cgroup for "memory.reclaim". This avoid the possible confusion there. >> > > > And, because demotion is considered aging, we don't need to disable >> > > > demotion for "memory.reclaim", just don't count it. >> > > >> > > Hm, so in summary: >> > > >> > > 1) memory.reclaim would demote and reclaim like today, but it would >> > > change to only count reclaimed pages against the goal. >> > > >> > > 2) memory.demote would only demote. >> > > >> >> If the above 2 points are agreeable then yes, this sounds good to me >> and does address our use case. >> >> > > a) What if the demotion targets are full? Would it reclaim or fail? >> > > >> >> Wei will chime in if he disagrees, but I think we _require_ that it >> fails, not falls back to reclaim. The interface is asking for >> demotion, and is called memory.demote. For such an interface to fall >> back to reclaim would be very confusing to userspace and may trigger >> reclaim on a high priority job that we want to shield from proactive >> reclaim. > > But what should happen if the immediate demotion target is full but > lower tiers are still usable. Should the first one demote before > allowing to demote from the top tier? > >> > > 3) Would memory.reclaim and memory.demote still need nodemasks? >> >> memory.demote will need a nodemask, for sure. Today the nodemask would >> be useful if there is a specific node in the top tier that is >> overloaded and we want to reduce the pressure by demoting. In the >> future there will be N tiers and the nodemask says which tier to >> demote from. > > OK, so what is the exact semantic of the node mask. Does it control > where to demote from or to or both? > >> I don't think memory.reclaim would need a nodemask anymore? At least I >> no longer see the use for it for us. >> >> > > Would >> > > they return -EINVAL if a) memory.reclaim gets passed only toptier >> > > nodes or b) memory.demote gets passed any lasttier nodes? >> > >> >> Honestly it would be great if memory.reclaim can force reclaim from a >> top tier nodes. It breaks the aginig pipeline, yes, but if the user is >> specifically asking for that because they decided in their usecase >> it's a good idea then the kernel should comply IMO. Not a strict >> requirement for us. Wei will chime in if he disagrees. > > That would require a nodemask to say which nodes to reclaim, no? The > default behavior should be in line with what standard memory reclaim > does. If the demotion is a part of that process so should be > memory.reclaim part of it. If we want to have a finer control then a > nodemask is really a must and then the nodemaks should constrain both > agining and reclaim. > >> memory.demote returning -EINVAL for lasttier nodes makes sense to me. >> >> > I would also add >> > 4) Do we want to allow to control the demotion path (e.g. which node to >> > demote from and to) and how to achieve that? >> >> We care deeply about specifying which node to demote _from_. That >> would be some node that is approaching pressure and we're looking for >> proactive saving from. So far I haven't seen any reason to control >> which nodes to demote _to_. The kernel deciding that based on the >> aging pipeline and the node distances sounds good to me. Obviously >> someone else may find that useful. > > Please keep in mind that the interface should be really prepared for > future extensions so try to abstract from your immediate usecases. I see two requirements here, one is to control the demotion source, that is, which nodes to free memory. The other is to control the demotion path. I think that we can use two different parameters for them, for example, "from=" and "to=". In most cases we don't need to control the demotion path. Because in current implementation, the nodes in the lower tiers in the same socket (local nodes) will be preferred. I think that this is the desired behavior in most cases. >> > 5) Is the demotion api restricted to multi-tier systems or any numa >> > configuration allowed as well? >> > >> >> demotion will of course not work on single tiered systems. The >> interface may return some failure on such systems or not be available >> at all. > > Is there any strong reason for that? We do not have any interface to > control NUMA balancing from userspace. Why cannot we use the interface > for that purpose? Do you mean to demote the cold pages from the specified source nodes to the specified target nodes in different sockets? We don't do that to avoid loop in the demotion path. If we prevent the target nodes from demoting cold pages to the source nodes at the same time, it seems doable. Best Regards, Huang, Ying