From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E73EAC4332F for ; Tue, 13 Dec 2022 15:58:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F97B8E0003; Tue, 13 Dec 2022 10:58:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 55A6D8E0002; Tue, 13 Dec 2022 10:58:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FB6B8E0003; Tue, 13 Dec 2022 10:58:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 29B108E0002 for ; Tue, 13 Dec 2022 10:58:59 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 90F0380732 for ; Tue, 13 Dec 2022 15:58:58 +0000 (UTC) X-FDA: 80237741556.19.521E51E Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by imf05.hostedemail.com (Postfix) with ESMTP id AC275100012 for ; Tue, 13 Dec 2022 15:58:56 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=tYSHyL3u; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.43 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670947137; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BmoHPjxBfuv0L4/bABJBapiPnoh74ZtmvdJi1uPpypQ=; b=KCYwZlcZvwt7BxVSpTgkBW8g0CaDjz5/wVBoEVG7pBJQjZRBgKLQXj8+Jr5euMU8LFYlta JokXFbmlRpTF8f06OLuSTpfuK+87bkDj2pGEHoU1q+jbZFoe97NiQWDNDx9wDNzbaGHwbW S3Nk1utHwU804VjJhdEPv+T326ChHb4= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=tYSHyL3u; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.43 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670947137; a=rsa-sha256; cv=none; b=HCVjBFQYlIM2WehU6cLgWix3DnFPg70DbM8libYh1Foz7RQ+NQkLw9R5sirHpufFR4DCeL B/Bcd4QAou8+2CxBGdwmGmYi3e2l5VIRsGk2mbgbXEstyshLUd8EFo7t8u2V4Ln6iSJ8vP l3dOM0tBpqYT/n1maj1RT/zUFAr4bOY= Received: by mail-qv1-f43.google.com with SMTP id c14so10739734qvq.0 for ; Tue, 13 Dec 2022 07:58:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=BmoHPjxBfuv0L4/bABJBapiPnoh74ZtmvdJi1uPpypQ=; b=tYSHyL3uT8BsXSP/ddwQ0c6U4nhdyNjgWj87JFBf3wKrv2/u1Rhvnbh0tcm5zQCdx+ Z3m6/3BDj8GObhrJtaTfOHOsj6JxliVkHsNt/y0O1l9MVpjQIn91+Sy3NrwwQeqcfZlQ S2PfH5LtOg/HYacxDBleObOg3/CePMpki5SMd7deB8e96YqDEQIAuxEAcxD2M+UZJxtA QqHXZx0xJUBuB9rXKD3WnIIY7Wo9FVQuTVEQnC4OudC5oSNZSw/P1OO6bf97dDABhUhK XjMDuG/ga6AuWOT2s47vBi+LPy3BfEM7s61d4FNSk/qXm+kDo6TDYl+nyQ5a4+92oU8M hROg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=BmoHPjxBfuv0L4/bABJBapiPnoh74ZtmvdJi1uPpypQ=; b=G1uPi15KRxlHMOFkLT2dA8ShLMxYGqqTHN60KNz4bTsuC9BV+huYNf84UnEX7UfS9x Aak1vsVMCNQKE4MSq0GP+7bxC4Q0LSABw15iKzqQaKAVulOj5cVl3jxoRbbeJg3GjuZ2 l5vcs7W/Tx500nj+D5JFofG1W0tDpe/FyZ+Gf+Hwn7F4xWEE1PueHiO5tZrzCgM1KPox 3IbTiAJ2TyRM0F4w/KjzuqqLiFSbZMM3LxJPnN9kvlzbPikIT/UMzNbwldPVHZyFQjt8 eaKB0yIRoVEEBc23UE5a1Ly0tE7ESFO9nNVlLiKkWx+PHamQvoHODOBtgnQ/2EJVHDYQ o6Dw== X-Gm-Message-State: ANoB5pnXBc295x1V0vXLY7l+SCSxBoICLcmfEzD2TMVgxuL5B1+KFs3g jhT2XDDOLQ0JUHshYAivLqshMA== X-Google-Smtp-Source: AA0mqf5TsTMn5+hH2JXzp36mG6kJIiORg6nfBCGE6ywjeW1MWc0UXgnjNCPVcFfhtcJ34w+X/VIqXg== X-Received: by 2002:a05:6214:2b8b:b0:4c7:82c7:8e12 with SMTP id kr11-20020a0562142b8b00b004c782c78e12mr29021586qvb.39.1670947135706; Tue, 13 Dec 2022 07:58:55 -0800 (PST) Received: from localhost ([107.116.82.98]) by smtp.gmail.com with ESMTPSA id bk24-20020a05620a1a1800b006ee77f1ecc3sm8067725qkb.31.2022.12.13.07.58.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Dec 2022 07:58:55 -0800 (PST) Date: Tue, 13 Dec 2022 16:58:50 +0100 From: Johannes Weiner To: Michal Hocko Cc: Mina Almasry , Tejun Heo , Zefan Li , Jonathan Corbet , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Huang Ying , Yang Shi , Yosry Ahmed , weixugc@google.com, fvdl@google.com, bagasdotme@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v3] mm: Add nodes= arg to memory.reclaim Message-ID: References: <20221202223533.1785418-1-almasrymina@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AC275100012 X-Stat-Signature: 6959ahn5dcf88zyrsu5egdr8ckjhbpyy X-Rspam-User: X-HE-Tag: 1670947136-359171 X-HE-Meta: U2FsdGVkX1+ikBzOtEkNWfnWZeOAbSUjDccKJQnyITdNUXJTn7cTE29E5VkpfKvnAxM7SkzC+ATo8ABS4taWMohlrhbmSPbWDAr0as4najHNpG6VCu+4cjKsSFbXKU64G7uYxAOaCXV7TTARdoOTuSkqmLW+mb5UgVVeLQRsXzsVW/OX9M1bMkiY+MWVqK51GScyL4RU5F3cGpZyrHF8YmT70NoRu3TNU/Rd8ee7b1d1mX5AJ69xL+OJ67O4gGagQUpxot/gu0RGvwniSdrwQAmIt7XLbLZyDkhJjMAXG5ZXnuhaT6K47VMrH6dAKKyxihqPOtyhZTIjoEleoyCvte3fi6h8gskOGmo7JQhjEoQYyE+RegZJ4sf+tf6zNHXxCDr19U+vLhM17XaamlbFZjT82egyFGlbObYbb9K+glH7BFPqyu3G3V/CwPuegweMMlX9KC3laHu0rEVqbJ1BQjYR9SxQLWY/zzUZehzEJ/6v9/3CVTliQ/j8DN9Vx/nfZzcalSA9L5xluNdz10yavy1WxVZ7SbmECI5o+rSQFLO9HP9i/FzoiC73KLBwvFfeg8NY3hQzTbd+XblgpRG3wMUORiMEHlvktU4xTEcsJb0KjxuSURuEsb7Rxs3N15RzE5bHN1TquW+AvKfo5f+r/0qxRuV6LeriuqdCoC+23h64rborJI1ZZXG+LhYcxxCoN7kYDj3bVD2EMcDuEcGr3oOw93ZYfq+pb3lSWx0V0SRWHQSZ0g6wzYAgte6jHmqfceyHVcol6aZle6a3rIqRTsCBxMdy0rZ5NIAiUScDGQIEDMIJCbE3AE9BQzLJqPIp52eUyYqC+WpUErwIujUGPAViJaMv2j9IQFp7HEPRwg4FM9nbIV8zI7cu1Fs/CRpdJNb51gOf0YDQXaXSwU4UGhxaYy4zR8aGZkAOm+n7RfsGBTDazBBo8NI/b2ea5xQM+/sgD5LnFPmGh7oRdVn qul5spQR jq/rFPYW20rVCjcqYv3yl32Yizroo9GChFreToLTM8CuiZ6wjixmkc6VKgb+5+CEFgS1dbrEBg+YWGFEd6d2/y7zsNn6FaJKQ0+WIflqLycUMRdu2j9KoxMfWaKqZgH4f+Za0vOyBrUktZ6nchd8WTxQuexL/Alnb+UEo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Dec 13, 2022 at 09:33:24AM +0100, Michal Hocko wrote: > I do recognize your need to control the demotion but I argue that it is > a bad idea to rely on an implicit behavior of the memory reclaim and an > interface which is _documented_ to primarily _reclaim_ memory. I think memory.reclaim should demote as part of page aging. What I'd like to avoid is *having* to manually control the aging component in the interface (e.g. making memory.reclaim *only* reclaim, and *requiring* a coordinated use of memory.demote to ensure progress.) > Really, consider that the current demotion implementation will change > in the future and based on a newly added heuristic memory reclaim or > compression would be preferred over migration to a different tier. This > might completely break your current assumptions and break your usecase > which relies on an implicit demotion behavior. Do you see that as a > potential problem at all? What shall we do in that case? Special case > memory.reclaim behavior? Shouldn't that be derived from the distance propertiers in the tier configuration? I.e. if local compression is faster than demoting to a slower node, we should maybe have a separate tier for that. Ignoring proactive reclaim or demotion commands for a second: on that node, global memory pressure should always compress first, while the oldest pages from the compression cache should demote to the other node(s) - until they eventually get swapped out. However fine-grained we make proactive reclaim control over these stages, it should at least be possible for the user to request the default behavior that global pressure follows, without jumping through hoops or requiring the coordinated use of multiple knobs. So IMO there is an argument for having a singular knob that requests comprehensive aging and reclaiming across the configured hierarchy. As far as explicit control over the individual stages goes - no idea if you would call the compression stage demotion or reclaim. The distinction still does not make much of sense to me, since reclaim is just another form of demotion. Sure, page faults have a different access latency than dax to slower memory. But you could also have 3 tiers of memory where the difference between tier 1 and 2 is much smaller than the difference between 2 and 3, and you might want to apply different demotion rates between them as well. The other argument is that demotion does not free cgroup memory, whereas reclaim does. But with multiple memory tiers of vastly different performance, isn't there also an argument for granting cgroups different shares of each memory? So that a higher priority group has access to a bigger share of the fastest memory, and lower prio cgroups are relegated to lower tiers. If we split those pools, then "demotion" will actually free memory in a cgroup. This is why I liked adding a nodes= argument to memory.reclaim the best. It doesn't encode a distinction that may not last for long. The problem comes from how to interpret the input argument and the return value, right? Could we solve this by requiring the passed nodes= to all be of the same memory tier? Then there is no confusion around what is requested and what the return value means. And if no nodes are passed, it means reclaim (from the lowest memory tier) X pages and demote as needed, then return the reclaimed pages. > Now to your specific usecase. If there is a need to do a memory > distribution balancing then fine but this should be a well defined > interface. E.g. is there a need to not only control demotion but > promotions as well? I haven't heard anybody requesting that so far > but I can easily imagine that like outsourcing the memory reclaim to > the userspace someone might want to do the same thing with the numa > balancing because $REASONS. Should that ever happen, I am pretty sure > hooking into memory.reclaim is not really a great idea. Should this ever happen, it would seem fair that that be a separate knob anyway, no? One knob to move the pipeline in one direction (aging), one knob to move it the other way.