From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B0C0C4321E for ; Wed, 30 Nov 2022 06:07:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 948AC6B0072; Wed, 30 Nov 2022 01:07:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F91F6B0073; Wed, 30 Nov 2022 01:07:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E73A6B0074; Wed, 30 Nov 2022 01:07:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6F8326B0072 for ; Wed, 30 Nov 2022 01:07:10 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 40CAF809F8 for ; Wed, 30 Nov 2022 06:07:10 +0000 (UTC) X-FDA: 80189075820.24.CC89F08 Received: from mail-ua1-f49.google.com (mail-ua1-f49.google.com [209.85.222.49]) by imf30.hostedemail.com (Postfix) with ESMTP id D128580009 for ; Wed, 30 Nov 2022 06:07:09 +0000 (UTC) Received: by mail-ua1-f49.google.com with SMTP id y15so5839752uan.6 for ; Tue, 29 Nov 2022 22:07:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=FWjVwsfaKDFRr8VkwYzgxCXQbcwfzKH8w2b9Cg9S3Qg=; b=G7O5bwbCEbvRUawdrMo0STCjIPBtikhsjYt5P2BbfdIgSX/6UUKrw+39wtAX3MzM3l asRuDPtvRAgh9vfobCy2BwwtQccULYWiaOcNFO0s4JiArEE6S4merDcjjJsdaJxZO0Lo uPyC5dNSHfSqryuTnefPXdCvzQOqfMVQT1Q5liZOs7a+r98V2iHJSORCQLhj5X1mLHFe MKbxTUXDzXCaXA12A8ibCEedbfC3hBGfeE9kq4l2Mn92IXBHHaT4/l2FjByD6fgQMBs1 hsWd6zuklbsRveIaFWwQa160MDjpU6F/UlFuI+0fLDn7OEOjdi3/Uh7m10etrOAQZAux ZHaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FWjVwsfaKDFRr8VkwYzgxCXQbcwfzKH8w2b9Cg9S3Qg=; b=4E0ksI4FsBfpodAxIXXW7kI5x6pdY3FhyT0BYA/J9MdetNjPw00vLrK052Ph/G1lMd 8IuzBQo9dPvjW9Ah76Gl8JRc/CgXpg7CxBXU3JuZSlzVmsXqPlDKBhOVlY3Tv7GVUb2s vDmeqhRbKNueQe8h4QgGjnapsCU/l1DHBvioLkixxrwCBFDReL6LZuJ2H5vosuhmAQ9f gX8nJsB9T81U9QNyFy6Yfum4dNrhnUoP946MtNleoIASlCz0SHDGlwZqPjlchgD6g3XH tcTdbNveDRm8cTZRuhi9C9dXN01TnxhNxp21Zp5S4DuYfOizsaRClrEpdEoOwlfYdLZQ K4Yw== X-Gm-Message-State: ANoB5pnRr0sKs47Ula0sZAB7YPOzrYhXeZ+vBGDbHQXzxkB4GRPSYLg/ BKD38J2+x5uX34ufNRTERRguh1V1TcSCCxSK3tSuNw== X-Google-Smtp-Source: AA0mqf6SVhumDSHrtQczqhp0r9DvE7CaXXHOLr0Yf/QvLnp1AFeQ0cn7D77MyLwPihqjlqNFkbRUGIrP2PaoMz6yH4c= X-Received: by 2002:ab0:5517:0:b0:409:5403:c18 with SMTP id t23-20020ab05517000000b0040954030c18mr26040308uaa.51.1669788428939; Tue, 29 Nov 2022 22:07:08 -0800 (PST) MIME-Version: 1.0 References: <20221122203850.2765015-1-almasrymina@google.com> <874juonbmv.fsf@yhuang6-desk2.ccr.corp.intel.com> <87edtlatmg.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: <87edtlatmg.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Mina Almasry Date: Tue, 29 Nov 2022 22:06:57 -0800 Message-ID: Subject: Re: [RFC PATCH V1] mm: Disable demotion from proactive reclaim To: "Huang, Ying" Cc: Johannes Weiner , Yang Shi , Yosry Ahmed , Tim Chen , weixugc@google.com, shakeelb@google.com, gthelen@google.com, fvdl@google.com, Michal Hocko , Roman Gushchin , Muchun Song , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669788429; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FWjVwsfaKDFRr8VkwYzgxCXQbcwfzKH8w2b9Cg9S3Qg=; b=76AGgvMQh1nAUNVuR9s1OHI37ImtnVE/6SW+esiGeIb7MvGDh7sAfOxoWhfaWG/i+dfodU fHmEVjOxa19xfjpOVN1A9+vu7VQyO4ZraxITQP9dwaQMKLYZnxiRVH+gZn9sy53XFwoOgn ECKglcX3IsY2eHUfoP6Yl9r7hne2Ko4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=G7O5bwbC; spf=pass (imf30.hostedemail.com: domain of almasrymina@google.com designates 209.85.222.49 as permitted sender) smtp.mailfrom=almasrymina@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669788429; a=rsa-sha256; cv=none; b=TOJHnYgJZrqKMMcVfu9p8uKKiptqzC17UKeGlfVOLjJ5xAGU+I1sl1STdNDSG+8mc+wuz4 VP1yKKhzlwvfOnmawBWc9IgDJb5NSJ2GZw72hA+wyXKeX21jtRIAQjGG2F0LRsHthBxLAO BKArcARxNDDREtwba9SSWdBwMROE0mI= X-Stat-Signature: 8op6oucc61xjyhmrgp11qh4ntkjar3ha X-Rspamd-Queue-Id: D128580009 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=G7O5bwbC; spf=pass (imf30.hostedemail.com: domain of almasrymina@google.com designates 209.85.222.49 as permitted sender) smtp.mailfrom=almasrymina@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1669788429-961657 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Nov 29, 2022 at 9:40 PM Huang, Ying wrote: > > Mina Almasry writes: > > > On Wed, Nov 23, 2022 at 9:52 PM Huang, Ying wrote: > >> > >> Hi, Johannes, > >> > >> Johannes Weiner writes: > >> [...] > >> > > >> > The fallback to reclaim actually strikes me as wrong. > >> > > >> > Think of reclaim as 'demoting' the pages to the storage tier. If we > >> > have a RAM -> CXL -> storage hierarchy, we should demote from RAM to > >> > CXL and from CXL to storage. If we reclaim a page from RAM, it means > >> > we 'demote' it directly from RAM to storage, bypassing potentially a > >> > huge amount of pages colder than it in CXL. That doesn't seem right. > >> > > >> > If demotion fails, IMO it shouldn't satisfy the reclaim request by > >> > breaking the layering. Rather it should deflect that pressure to the > >> > lower layers to make room. This makes sure we maintain an aging > >> > pipeline that honors the memory tier hierarchy. > >> > >> Yes. I think that we should avoid to fall back to reclaim as much as > >> possible too. Now, when we allocate memory for demotion > >> (alloc_demote_page()), __GFP_KSWAPD_RECLAIM is used. So, we will trigger > > > > I may be missing something but as far I can tell reclaim is disabled > > for allocations from lower tier memory: > > https://elixir.bootlin.com/linux/v6.1-rc7/source/mm/vmscan.c#L1583 > > #define GFP_NOWAIT (__GFP_KSWAPD_RECLAIM) > > We have GFP_NOWAIT set in gfp. > Ah, thanks. I missed that. > > I think this is maybe a good thing when doing proactive demotion. In > > this case we probably don't want to try to reclaim from lower tier > > nodes and instead fail the proactive demotion. > > Do you have some real use cases for this? If so, we can tweak the > logic. > Nothing real at the moment. I was thinking this may be something desirable to tune at some point. > > However I can see this being desirable when the top tier nodes are > > under real memory pressure to deflect that pressure to the lower tier > > nodes. > > Yes. > > Best Regards, > Huang, Ying > > >> kswapd reclaim on lower tier node to free some memory to avoid fall back > >> to reclaim on current (higher tier) node. This may be not good enough, > >> for example, the following patch from Hasan may help via waking up > >> kswapd earlier. > >> > >> https://lore.kernel.org/linux-mm/b45b9bf7cd3e21bca61d82dcd1eb692cd32c122c.1637778851.git.hasanalmaruf@fb.com/ > >> > >> Do you know what is the next step plan for this patch? > >> > >> Should we do even more? > >> > >> From another point of view, I still think that we can use falling back > >> to reclaim as the last resort to avoid OOM in some special situations, > >> for example, most pages in the lowest tier node are mlock() or too hot > >> to be reclaimed. > >> > >> > So I'm hesitant to design cgroup controls around the current behavior. > > > > I sent RFC v2 patch: > > https://lore.kernel.org/linux-mm/20221130020328.1009347-1-almasrymina@google.com/T/#u > > > > Please take a look when convenient. Thanks! > > > >> > > >> > >> Best Regards, > >> Huang, Ying > >> >