From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F767C3A5A7 for ; Thu, 1 Dec 2022 22:45:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C9656B0073; Thu, 1 Dec 2022 17:45:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 07A466B0074; Thu, 1 Dec 2022 17:45:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5BDD6B0075; Thu, 1 Dec 2022 17:45:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D416B6B0073 for ; Thu, 1 Dec 2022 17:45:50 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A5526ABB25 for ; Thu, 1 Dec 2022 22:45:50 +0000 (UTC) X-FDA: 80195221260.23.A351CD5 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf14.hostedemail.com (Postfix) with ESMTP id 4F1BB10000F for ; Thu, 1 Dec 2022 22:45:49 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Oxk08d7L; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=shy828301@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669934749; a=rsa-sha256; cv=none; b=NZNQIArfzDh/FeJU4apZJzlvR1rfZDqXiyzv+xhwO9si3juUXuMx+1JtpnNd8X5yqhYRlR iWuYF4yHydVUcw8vuZpIHO8XZCYDonXjcugsQMBPSyjNmAiyOAfu7W62AVgL0xZ9/gHRVf lPT06SSyskn4bEvU4koMVpnFNU2yv3k= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Oxk08d7L; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=shy828301@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669934749; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mz67COhiP2LIOSR/RjOjcqP515AruE68ATdaRH6nni8=; b=XmvyZk/WyVvLlcG0PjIbwrp1UQis944OCDIm/i50Vl2c/lPxD1H+cgOqf+8U+NXmlWF+hE 2O5p1HiCvYncCKsbw0zB7TNO9If0m6qnmpruvzCL4tw2aBA5735erDHjyKj5gFKoiU3Wvo QCzVOasgSx+4n9BqP8sn3FAuCsWOelw= Received: by mail-pf1-f179.google.com with SMTP id h28so3241111pfq.9 for ; Thu, 01 Dec 2022 14:45:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=mz67COhiP2LIOSR/RjOjcqP515AruE68ATdaRH6nni8=; b=Oxk08d7LgKIO11W2YXoWDp6N77NPHo4awIEjlThe1KatRgbufsRdv1EOloK/sIsqzO jvIWN4M6ORdZmL0nk+OVwMmcad0Rll39eegzg+hXKs3DWJCwiwRC7Apml/RQX9mnbhQT YQ0yY1ACQDY5J6Qp3qijLh/GR5Z66lQgq3jen0fIAefZ7/Llw+nG5ln/hG4Qf/6o9fPn ieT9ql1qAQeNzDPTf0hDy2ZQ+pfV340gYX3gej01B1a+ivcoxfZTSovz4uWzOgnYiX5/ FC9ENBqRNG1JLaU/biOmKGDMFqd24XHZ03OEhFC1SGTm8vhxWO1mYO5Evicn9TAUY+eO 2uIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mz67COhiP2LIOSR/RjOjcqP515AruE68ATdaRH6nni8=; b=7DI6x7y0eRMXLx56IMq9kxyTfRuEnmHA+rgKnvrer1Kbrw1djf8vTj6C9Ba8bMO3NM 68qwJbn1jQpNxmB1IZYtE7k+8KanjA+OD8PSH8gc0Sh8nKLAP5wLbvU0cI+bDylPQjVc 0Sk6wSyUUWP2ApFE9P1dW//VBzCnJh332liOCtuoOcEW6J2RtDSbiEByJqdg5YdTcS4k NxJdJjx2Mf2V4xoze5x/iWUkeq5oCIfgSy8muzHvxlQr5Kbevq4itP1MigA81S6U5DvG SD4A6/Z+qnoUKL+IjwnmjXGKl6hPWecZnyi0+soshL+hocoSFh+hMr3897Wob/q9q+wu lc5w== X-Gm-Message-State: ANoB5plBwdBzZYz4W5siVyrknYPpPuH+FLZ7ivEwjFTgwC3do6zKblkz Ti+oE85Q8V+wPJRuVw1DqnQmC0EU0/yoj32R7vI= X-Google-Smtp-Source: AA0mqf44AXvVEUJV0E8tjOZWmdq5g2PebFd55FpDXlWOKp5M3htocvHLrqDql5SJAScaZGwcfDBNydfMw3AH9ZFYC/s= X-Received: by 2002:a63:105c:0:b0:46f:f72c:cdae with SMTP id 28-20020a63105c000000b0046ff72ccdaemr44554126pgq.237.1669934748081; Thu, 01 Dec 2022 14:45:48 -0800 (PST) MIME-Version: 1.0 References: <20221122203850.2765015-1-almasrymina@google.com> <874juonbmv.fsf@yhuang6-desk2.ccr.corp.intel.com> <87a64ad1iz.fsf@yhuang6-desk2.ccr.corp.intel.com> <87ilixatyw.fsf@yhuang6-desk2.ccr.corp.intel.com> <87h6yfao37.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: <87h6yfao37.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Yang Shi Date: Thu, 1 Dec 2022 14:45:36 -0800 Message-ID: Subject: Re: [RFC PATCH V1] mm: Disable demotion from proactive reclaim To: "Huang, Ying" Cc: Johannes Weiner , Mina Almasry , Yang Shi , Yosry Ahmed , Tim Chen , weixugc@google.com, shakeelb@google.com, gthelen@google.com, fvdl@google.com, Michal Hocko , Roman Gushchin , Muchun Song , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 4F1BB10000F X-Stat-Signature: tmossgm5usrudteisya5j6xmf8orybbg X-Spamd-Result: default: False [0.10 / 9.00]; SORBS_IRL_BL(3.00)[209.85.210.179:from]; BAYES_HAM(-3.00)[100.00%]; BAD_REP_POLICIES(0.10)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DMARC_POLICY_ALLOW(0.00)[gmail.com,none]; RCPT_COUNT_TWELVE(0.00)[17]; DKIM_TRACE(0.00)[gmail.com:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; FROM_HAS_DN(0.00)[]; R_DKIM_ALLOW(0.00)[gmail.com:s=20210112]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17:c]; ARC_NA(0.00)[] X-Rspam-User: X-HE-Tag: 1669934749-391794 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Nov 30, 2022 at 5:52 PM Huang, Ying wrote: > > Yang Shi writes: > > > On Tue, Nov 29, 2022 at 9:33 PM Huang, Ying wrote: > >> > >> Yang Shi writes: > >> > >> > On Mon, Nov 28, 2022 at 4:54 PM Huang, Ying wrote: > >> >> > >> >> Yang Shi writes: > >> >> > >> >> > On Wed, Nov 23, 2022 at 9:52 PM Huang, Ying wrote: > >> >> >> > >> >> >> Hi, Johannes, > >> >> >> > >> >> >> Johannes Weiner writes: > >> >> >> [...] > >> >> >> > > >> >> >> > The fallback to reclaim actually strikes me as wrong. > >> >> >> > > >> >> >> > Think of reclaim as 'demoting' the pages to the storage tier. If we > >> >> >> > have a RAM -> CXL -> storage hierarchy, we should demote from RAM to > >> >> >> > CXL and from CXL to storage. If we reclaim a page from RAM, it means > >> >> >> > we 'demote' it directly from RAM to storage, bypassing potentially a > >> >> >> > huge amount of pages colder than it in CXL. That doesn't seem right. > >> >> >> > > >> >> >> > If demotion fails, IMO it shouldn't satisfy the reclaim request by > >> >> >> > breaking the layering. Rather it should deflect that pressure to the > >> >> >> > lower layers to make room. This makes sure we maintain an aging > >> >> >> > pipeline that honors the memory tier hierarchy. > >> >> >> > >> >> >> Yes. I think that we should avoid to fall back to reclaim as much as > >> >> >> possible too. Now, when we allocate memory for demotion > >> >> >> (alloc_demote_page()), __GFP_KSWAPD_RECLAIM is used. So, we will trigger > >> >> >> kswapd reclaim on lower tier node to free some memory to avoid fall back > >> >> >> to reclaim on current (higher tier) node. This may be not good enough, > >> >> >> for example, the following patch from Hasan may help via waking up > >> >> >> kswapd earlier. > >> >> > > >> >> > For the ideal case, I do agree with Johannes to demote the page tier > >> >> > by tier rather than reclaiming them from the higher tiers. But I also > >> >> > agree with your premature OOM concern. > >> >> > > >> >> >> > >> >> >> https://lore.kernel.org/linux-mm/b45b9bf7cd3e21bca61d82dcd1eb692cd32c122c.1637778851.git.hasanalmaruf@fb.com/ > >> >> >> > >> >> >> Do you know what is the next step plan for this patch? > >> >> >> > >> >> >> Should we do even more? > >> >> > > >> >> > In my initial implementation I implemented a simple throttle logic > >> >> > when the demotion is not going to succeed if the demotion target has > >> >> > not enough free memory (just check the watermark) to make migration > >> >> > succeed without doing any reclamation. Shall we resurrect that? > >> >> > >> >> Can you share the link to your throttle patch? Or paste it here? > >> > > >> > I just found this on the mailing list. > >> > https://lore.kernel.org/linux-mm/1560468577-101178-8-git-send-email-yang.shi@linux.alibaba.com/ > >> > >> Per my understanding, this patch will avoid demoting if there's no free > >> space on demotion target? If so, I think that we should trigger kswapd > >> reclaiming on demotion target before that. And we can simply avoid to > >> fall back to reclaim firstly, then avoid to scan as an improvement as > >> that in your patch above. > > > > Yes, it should. The rough idea looks like: > > > > if (the demote target is contended) > > wake up kswapd > > reclaim_throttle(VMSCAN_THROTTLE_DEMOTION) > > retry demotion > > > > The kswapd is responsible for clearing the contention flag. > > We may do this, at least for demotion in kswapd. But I think that this > could be the second step optimization after we make correct choice > between demotion/reclaim. What if the pages in demotion target is too > hot to be reclaimed first? Should we reclaim in fast memory node to > avoid OOM? IMHO we can't avoid reclaiming from the fast nodes entirely if we prioritize avoiding OOMs. But it should happen very very rarely with the throttling logic or other methods. BTW did you run any test to see how many times vmscan reclaims from fast nodes instead of demotion with the current implementation for some typical workloads? > > Best Regards, > Huang, Ying > > >> > >> > But it didn't have the throttling logic, I may not submit that version > >> > to the mailing list since we decided to drop this and merge mine and > >> > Dave's. > >> > > >> > Anyway it is not hard to add the throttling logic, we already have a > >> > few throttling cases in vmscan, for example, "mm/vmscan: throttle > >> > reclaim until some writeback completes if congested". > >> >> > >> >> > Waking kswapd sooner is fine to me, but it may be not enough, for > >> >> > example, the kswapd may not keep up so remature OOM may happen on > >> >> > higher tiers or reclaim may still happen. I think throttling the > >> >> > reclaimer/demoter until kswapd makes progress could avoid both. And > >> >> > since the lower tiers memory typically is quite larger than the higher > >> >> > tiers, so the throttle should happen very rarely IMHO. > >> >> > > >> >> >> > >> >> >> From another point of view, I still think that we can use falling back > >> >> >> to reclaim as the last resort to avoid OOM in some special situations, > >> >> >> for example, most pages in the lowest tier node are mlock() or too hot > >> >> >> to be reclaimed. > >> >> >> > >> >> >> > So I'm hesitant to design cgroup controls around the current behavior. > >> >> > >> >> Best Regards, > >> >> Huang, Ying