From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A899C4332F for ; Wed, 14 Dec 2022 10:43:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C49948E0005; Wed, 14 Dec 2022 05:43:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BFB4E8E0002; Wed, 14 Dec 2022 05:43:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC1A88E0005; Wed, 14 Dec 2022 05:43:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9D3748E0002 for ; Wed, 14 Dec 2022 05:43:24 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5F27D160365 for ; Wed, 14 Dec 2022 10:43:24 +0000 (UTC) X-FDA: 80240575128.05.FF30CE2 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf12.hostedemail.com (Postfix) with ESMTP id 9E05C40002 for ; Wed, 14 Dec 2022 10:43:22 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=tjmQwSC5; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf12.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671014602; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pOLj6orCcUIQaK1bmDnm+LNeXET6CuRHzqb6smtqDhg=; b=M2zVxeiOOyrhDNgwut9d/7eruyoj9uVKFkbID9tzS+2Pk1R6B7a/DIxoqWv8JHLdbHJMKy Nf+XFnxWnK4rvCcW8TJDNsOcOEn/Yr0NJZS210pVszMlbAYTJKbX5WPSQEIfww5R2F/aXy 3IHOhQ1W13ugxpytOwj4gXrrqeocEIk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=tjmQwSC5; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf12.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671014602; a=rsa-sha256; cv=none; b=vt5m5sx95ZlimLSDkGMY6gyUyduXRjqpUPRAs3gy1R0YMtpgf8c11nhLECgJ7V74kCDtJh 6RtzNKvrxWt4x1YQ77DQTmTqyHW+Z3paeaWADlebuVao1Fv9E6SeQwHl5utB4XXZzo1xne T9oM/qIp9mUexFS+tQVpn8v1Jic95SQ= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 202AB22181; Wed, 14 Dec 2022 10:43:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1671014601; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=pOLj6orCcUIQaK1bmDnm+LNeXET6CuRHzqb6smtqDhg=; b=tjmQwSC5l46EbQoMLj96F7w8R2nfE0/DWBVttEI5sO7CPV/4mkrC5QCI+5wL64xI56tI7L 5hAXErj/GwBzqruajdRn2lNn/w4NwtVCIjzOJxpqurEpCJwoK24hFs04eu91K98KkKeLZ+ B2dlnycoWRyJ6RfyHL2Jp/tyTWWG/KA= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id EB6B61333E; Wed, 14 Dec 2022 10:43:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id lEwKN8iomWPJKQAAMHmgww (envelope-from ); Wed, 14 Dec 2022 10:43:20 +0000 Date: Wed, 14 Dec 2022 11:43:20 +0100 From: Michal Hocko To: Johannes Weiner Cc: Mina Almasry , Tejun Heo , Zefan Li , Jonathan Corbet , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Huang Ying , Yang Shi , Yosry Ahmed , weixugc@google.com, fvdl@google.com, bagasdotme@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v3] mm: Add nodes= arg to memory.reclaim Message-ID: References: <20221202223533.1785418-1-almasrymina@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9E05C40002 X-Stat-Signature: ex51616kk33t5hh9649je3kufcz7od7q X-HE-Tag: 1671014602-313494 X-HE-Meta: U2FsdGVkX1/CtFhpHmifZWOYPxzFHoyK77hQjBBQ1eDvs1BhkbcoF07cw2w7Be9VRDvBfVEltjFakiwclOExRsOXTzU6FvWf4nkBRRfKfEniWQJ2IRljcbBTgFmhKva5V/h5C2AiWvVr0vyxoUK8+cUcFqzgfqiXd2fEESbHjatj14jrQcVDS3EHOeyWcqI6hl+CduF2cXnofkbAKCNM2Bcw08wxLq+uH2ca5cC/ovTltJTx3Zggk87s+pYaLZxsuD3/6t/CimS/pFGaInvyLc5i6Ghb+0wvbpIt/q67AJqZgjObygDvBje+qIBKhAoQFGnMKZGFAdYtFg/1x+6tDS1gagRj0de8uTlZn+3OLHGC2SscNbabEp2ClbAgo3QyzDZ8sxMF5q7JRJXomgu2vS/BfJLGmSN+3bpLU+p2wslfV6fus9J34dgujKG3fRXF8+mqSj72Svpl9TSQBNE+I16Bm8hoAVtQry6yszFX1dZLbVTIj2LUyLv4zIVhp7YEgNSfSR7kgSJVUH9u24n6E6v2O3DnhaOlghggUtE3nLEzeF5P3yoyIdF5A6Ul0URlD9bBAxhw4/ajbJY8mtw8IttkHsnGlfeudXf9aXX8LWlfJwvj941zgrObnXf9gBW0Fw1aI7Vcv3US7/f8/BxJjQjgdakuzaEEA+Ua2bMOf85vss6Dm3aTEDUyCSpcVwhn72h34Qxichs8Vc3mxG7kPi62kuF/5h/SWNi6Y4P5kvVBYQ0iSuKlZMX0ZRJrQoGYqmOd+iq7Muq6odOAV8eGtppLtlWQ/k/Baac9GmegMhjjDOcgrS6PtazlPvt94Qu18FbZhnYyiAXJaleuLCd7m62TBbZetGfxfjAaHHPrHi/jhCQDk8cOBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue 13-12-22 16:58:50, Johannes Weiner wrote: > On Tue, Dec 13, 2022 at 09:33:24AM +0100, Michal Hocko wrote: > > I do recognize your need to control the demotion but I argue that it is > > a bad idea to rely on an implicit behavior of the memory reclaim and an > > interface which is _documented_ to primarily _reclaim_ memory. > > I think memory.reclaim should demote as part of page aging. What I'd > like to avoid is *having* to manually control the aging component in > the interface (e.g. making memory.reclaim *only* reclaim, and > *requiring* a coordinated use of memory.demote to ensure progress.) Yes, I do agree with that. Demotion is a part of the aging. I meant to say that the result of the operation should be reclaimed charges but that doesn't mean that demotion is not a part of that process. I am mostly concerned about demote only behavior that Mina is targetting and want to use memory.reclaim interface. > > Really, consider that the current demotion implementation will change > > in the future and based on a newly added heuristic memory reclaim or > > compression would be preferred over migration to a different tier. This > > might completely break your current assumptions and break your usecase > > which relies on an implicit demotion behavior. Do you see that as a > > potential problem at all? What shall we do in that case? Special case > > memory.reclaim behavior? > > Shouldn't that be derived from the distance propertiers in the tier > configuration? > > I.e. if local compression is faster than demoting to a slower node, we > should maybe have a separate tier for that. Ignoring proactive reclaim > or demotion commands for a second: on that node, global memory > pressure should always compress first, while the oldest pages from the > compression cache should demote to the other node(s) - until they > eventually get swapped out. > > However fine-grained we make proactive reclaim control over these > stages, it should at least be possible for the user to request the > default behavior that global pressure follows, without jumping through > hoops or requiring the coordinated use of multiple knobs. So IMO there > is an argument for having a singular knob that requests comprehensive > aging and reclaiming across the configured hierarchy. > > As far as explicit control over the individual stages goes - no idea > if you would call the compression stage demotion or reclaim. The > distinction still does not make much of sense to me, since reclaim is > just another form of demotion. >From the external visibility POV the major difference between the two is that the reclaim decreases the overall charged memory. And there are pro-active reclaim usecases which rely on that. Demotion is mostly memory placement rebalancing. Sure still visible in per-node stats and with implications to performance but that is a different story. > Sure, page faults have a different > access latency than dax to slower memory. But you could also have 3 > tiers of memory where the difference between tier 1 and 2 is much > smaller than the difference between 2 and 3, and you might want to > apply different demotion rates between them as well. > > The other argument is that demotion does not free cgroup memory, > whereas reclaim does. But with multiple memory tiers of vastly > different performance, isn't there also an argument for granting > cgroups different shares of each memory? Yes. We have already had requests for per node limits in the past. And I do expect this will show up as a problem here as well but with a reasonable memory.reclaim and potentially memory.demote interfaces the balancing and policy making can be outsourced to the userspace . > So that a higher priority > group has access to a bigger share of the fastest memory, and lower > prio cgroups are relegated to lower tiers. If we split those pools, > then "demotion" will actually free memory in a cgroup. > > This is why I liked adding a nodes= argument to memory.reclaim the > best. It doesn't encode a distinction that may not last for long. > > The problem comes from how to interpret the input argument and the > return value, right? Could we solve this by requiring the passed > nodes= to all be of the same memory tier? Then there is no confusion > around what is requested and what the return value means. Just to make sure I am on the same page. This means that if a node mask is specified then it always implies demotion without any control over how the demotion is done, right? > And if no nodes are passed, it means reclaim (from the lowest memory > tier) X pages and demote as needed, then return the reclaimed pages. IMO this is rather constrained semantic which will completely rule out some potentially interesting usecases. E.g. fine grained control over the demotion path or enforced reclaim for node balancing. Also if we ever want a promote interface then it would better fit with demote counterpart. > > Now to your specific usecase. If there is a need to do a memory > > distribution balancing then fine but this should be a well defined > > interface. E.g. is there a need to not only control demotion but > > promotions as well? I haven't heard anybody requesting that so far > > but I can easily imagine that like outsourcing the memory reclaim to > > the userspace someone might want to do the same thing with the numa > > balancing because $REASONS. Should that ever happen, I am pretty sure > > hooking into memory.reclaim is not really a great idea. > > Should this ever happen, it would seem fair that that be a separate > knob anyway, no? One knob to move the pipeline in one direction > (aging), one knob to move it the other way. Yes, this is what I am inclining to as well. -- Michal Hocko SUSE Labs