From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E096C433FE for ; Wed, 23 Nov 2022 21:57:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C34F6B0071; Wed, 23 Nov 2022 16:57:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 473B26B0072; Wed, 23 Nov 2022 16:57:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33B036B0074; Wed, 23 Nov 2022 16:57:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 239D76B0071 for ; Wed, 23 Nov 2022 16:57:46 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DDF6A8043D for ; Wed, 23 Nov 2022 21:57:45 +0000 (UTC) X-FDA: 80166069690.13.16D00ED Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf28.hostedemail.com (Postfix) with ESMTP id 334E4C0009 for ; Wed, 23 Nov 2022 21:57:45 +0000 (UTC) Received: by mail-qt1-f169.google.com with SMTP id cg5so64326qtb.12 for ; Wed, 23 Nov 2022 13:57:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=eXUl97TIC36iX2gzZ4z9T7kVADet1dCMIeHKCLo4dC4=; b=1d778WwTHiFnSrOOtipV1oy8hteglhzctqXDcakElk44hR7nj0Kw1HZUKFrkLK1flS 1dy2ejEUeQqR3zUjNbKe64kHFHaCeNPfR+mHEFOOM8LiucYx2e8pjbjLk1BMo3Sbf3K9 wZL/Jay42f6APIg1zJmYPEYEFZuFaYZ54oLip8A+55aEI0PiSRx6eLERACZ4gNoB/nRm LTOQd16sAxmRdLvYL+ViYyxYH5UhhXYq81mueLRa3i7YmQ4Cty/v68jpsO7pFl5W2ApC fFUnjjrFHbN4SCxNJgdTGJifAWOycqM5/tm/mDgGyEo4EcDqRaXaB8j0C8GsSVJo9CBj 8CpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=eXUl97TIC36iX2gzZ4z9T7kVADet1dCMIeHKCLo4dC4=; b=dxpfyEVxzUx3pdpp3m/xDUmbYgfm1ua6CfkZHlmsk3kn2jtScis39PUz+QmAftX6AB KGf5/FjvG5nApkHRnTGI46r7t4qagMIUgYkPmO3wZaDlq6xUVNgunaHgZ+95BQ7ljQun +9cTNnL9piL0qjZ2hXXO36lVy6JC9en4Z/QIrQpWZiOLsYp4CJHNK+BBTGr9AytM8D6K VeCLQqeNLJ6HBLAEKLlFJ+bnlfn2my8OFZx4ZKkdGknCc34du6bZqgcpuZahHjFzJ4cj GGYajuayov33AtWP92U/hA9UTIzKUPg7P8fLmhEOBzGU7oycetD3R5dAEVcz6OEaUR8x vOrQ== X-Gm-Message-State: ANoB5plvCm7B5ww9by/tX8laPG3u/vINmdv5sOqqkWPROz7pFiVn1wBj OsQ46q7g1ctqc7FZjjlcZKcrbw== X-Google-Smtp-Source: AA0mqf59P97n8g2DJvpHoXjjiYjLodaojFr8pNiZ3Pf2y67xR6vLpb15b1401HTaq887RBMqTNLCfg== X-Received: by 2002:ac8:4a10:0:b0:3a5:2556:42ed with SMTP id x16-20020ac84a10000000b003a5255642edmr13309728qtq.422.1669240664291; Wed, 23 Nov 2022 13:57:44 -0800 (PST) Received: from localhost ([2620:10d:c091:480::1:bc4]) by smtp.gmail.com with ESMTPSA id t18-20020a05620a451200b006fafc111b12sm13219537qkp.83.2022.11.23.13.57.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Nov 2022 13:57:43 -0800 (PST) Date: Wed, 23 Nov 2022 16:58:10 -0500 From: Johannes Weiner To: Mina Almasry Cc: Huang Ying , Yang Shi , Yosry Ahmed , Tim Chen , weixugc@google.com, shakeelb@google.com, gthelen@google.com, fvdl@google.com, Michal Hocko , Roman Gushchin , Muchun Song , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: Re: [RFC PATCH V1] mm: Disable demotion from proactive reclaim Message-ID: References: <20221122203850.2765015-1-almasrymina@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669240665; a=rsa-sha256; cv=none; b=kwC3tYQjmtjEv7qeu15/P+EcQmmihWtF9hgiu9X9KyRKoSUD5EJVVUDBp3Efuez3GwzHPt SjY38nPLtbVdJOlT9HXrdP/xEL8OVpNcZRI5HYpM+mrHpn+GRu/LcaHmTCXEPBMDEFHjLW 0e8T+t98jUlMGgVN9R/7j9+A8OvE9sg= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=1d778WwT; spf=pass (imf28.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.169 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669240665; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eXUl97TIC36iX2gzZ4z9T7kVADet1dCMIeHKCLo4dC4=; b=dpY89evvVVk87zntjB4PjtMbcZyUa54XOM8/1sQL+ttWQ8VuMWqRGb4sHs8WMy4AaUFjwH lgurJ8K/faKfSqYedJHGZmrKh9hRA5HpTBj0CMyuLrqb8qKO0NUow3L33w9qstBMj0DfjG U38awLQgC+Dtw9ffdnVXN0TxvndFnNE= X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 334E4C0009 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=1d778WwT; spf=pass (imf28.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.169 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Rspam-User: X-Stat-Signature: 8hikw4jd6qsbc3xgoicgwac44utc9nwg X-HE-Tag: 1669240664-487383 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Nov 23, 2022 at 01:20:57PM -0800, Mina Almasry wrote: > On Wed, Nov 23, 2022 at 10:00 AM Johannes Weiner wrote: > > > > Hello Mina, > > > > On Tue, Nov 22, 2022 at 12:38:45PM -0800, Mina Almasry wrote: > > > Since commit 3f1509c57b1b ("Revert "mm/vmscan: never demote for memcg > > > reclaim""), the proactive reclaim interface memory.reclaim does both > > > reclaim and demotion. This is likely fine for us for latency critical > > > jobs where we would want to disable proactive reclaim entirely, and is > > > also fine for latency tolerant jobs where we would like to both > > > proactively reclaim and demote. > > > > > > However, for some latency tiers in the middle we would like to demote but > > > not reclaim. This is because reclaim and demotion incur different latency > > > costs to the jobs in the cgroup. Demoted memory would still be addressable > > > by the userspace at a higher latency, but reclaimed memory would need to > > > incur a pagefault. > > > > > > To address this, I propose having reclaim-only and demotion-only > > > mechanisms in the kernel. There are a couple possible > > > interfaces to carry this out I considered: > > > > > > 1. Disable demotion in the memory.reclaim interface and add a new > > > demotion interface (memory.demote). > > > 2. Extend memory.reclaim with a "demote=" flag to configure the demotion > > > behavior in the kernel like so: > > > - demote=0 would disable demotion from this call. > > > - demote=1 would allow the kernel to demote if it desires. > > > - demote=2 would only demote if possible but not attempt any > > > other form of reclaim. > > > > Unfortunately, our proactive reclaim stack currently relies on > > memory.reclaim doing both. It may not stay like that, but I'm a bit > > wary of changing user-visible semantics post-facto. > > > > In patch 2, you're adding a node interface to memory.demote. Can you > > add this to memory.reclaim instead? This would allow you to control > > demotion and reclaim independently as you please: if you call it on a > > node with demotion targets, it will demote; if you call it on a node > > without one, it'll reclaim. And current users will remain unaffected. > > Hello Johannes, thanks for taking a look! > > I can certainly add the "nodes=" arg to memory.reclaim and you're > right, that would help in bridging the gap. However, if I understand > the underlying code correctly, with only the nodes= arg the kernel > will indeed attempt demotion first, but the kernel will also merrily > fall back to reclaiming if it can't demote the full amount. I had > hoped to have the flexibility to protect latency sensitive jobs from > reclaim entirely while attempting to do demotion. The fallback to reclaim actually strikes me as wrong. Think of reclaim as 'demoting' the pages to the storage tier. If we have a RAM -> CXL -> storage hierarchy, we should demote from RAM to CXL and from CXL to storage. If we reclaim a page from RAM, it means we 'demote' it directly from RAM to storage, bypassing potentially a huge amount of pages colder than it in CXL. That doesn't seem right. If demotion fails, IMO it shouldn't satisfy the reclaim request by breaking the layering. Rather it should deflect that pressure to the lower layers to make room. This makes sure we maintain an aging pipeline that honors the memory tier hierarchy. So I'm hesitant to design cgroup controls around the current behavior. > The above is just one angle of the issue. Another angle (which Yosry > would care most about I think) is that at Google we call > memory.reclaim mainly when memory.current is too close to memory.max > and we expect the memory usage of the cgroup to drop as a result of a > success memory.reclaim call. I suspect once we take in commit > 3f1509c57b1b ("Revert "mm/vmscan: never demote for memcg reclaim""), > we would run into that regression, but I defer to Yosry here, he may > have a solution for that in mind already. IMO it should both demote and reclaim. Simliar to how memory.reclaim on a non-tiered memory system would both deactivate active pages and reclaim inactive pages.