From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2DECC4332F for ; Fri, 2 Dec 2022 02:06:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A9C66B0071; Thu, 1 Dec 2022 21:06:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 359BA6B0073; Thu, 1 Dec 2022 21:06:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F9CB6B0074; Thu, 1 Dec 2022 21:06:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0D48A6B0071 for ; Thu, 1 Dec 2022 21:06:32 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C74521C6BCB for ; Fri, 2 Dec 2022 02:06:31 +0000 (UTC) X-FDA: 80195726982.05.648C2FA Received: from mail-vs1-f54.google.com (mail-vs1-f54.google.com [209.85.217.54]) by imf24.hostedemail.com (Postfix) with ESMTP id 7CC8618000F for ; Fri, 2 Dec 2022 02:06:31 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FGvlzpkl; spf=pass (imf24.hostedemail.com: domain of almasrymina@google.com designates 209.85.217.54 as permitted sender) smtp.mailfrom=almasrymina@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669946791; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1N+8R3B0R8scfqdo5hcRRcXMxMacRIFBiFcRxzf5Jso=; b=OoTkzO6mLoq8SZ3abbFNG24HcMv7tu6eWK9BRXaxIrEBxq0BrW8gVDq9frVGSt4iCnkAmZ zGAdZZIXJh+F80Dtx1I7KKBHAi+APWvsNTK5iMibZwW2AhY0/qvOe8n+XR5Xu5jJ6b0AkY TGalCiDP2JaiaLJdCZK4oApVRTPd6Og= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669946791; a=rsa-sha256; cv=none; b=4VY3TT8tHDRIrTSAAIgfBgJdRDdZzzR/bLHm5jvSIUjo20G5/nGTjTbtM3lXQfovWMWNzB kF/UR4n0yzZEAHA+GQyLYCI3uGpC2IK6wUyUxVIYFFyD8mlaORHM4aISNcgtVwQobBBgyG JBvZS+jvJ/WaYIQhW/u6yO9+mPfif/c= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FGvlzpkl; spf=pass (imf24.hostedemail.com: domain of almasrymina@google.com designates 209.85.217.54 as permitted sender) smtp.mailfrom=almasrymina@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-vs1-f54.google.com with SMTP id 3so2551834vsq.7 for ; Thu, 01 Dec 2022 18:06:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=1N+8R3B0R8scfqdo5hcRRcXMxMacRIFBiFcRxzf5Jso=; b=FGvlzpklux1eQvCCAhLE2Dtsmq6kxEJYMyrDPhWhi4Ffz08/w1SdCuiWgYdyoClJrY a0R+QzFLk3lpRqYN6QRoKAZl3xnbErPu1BCj04OFhGMBvJ7ekKm6VjmUMppeSXtW/nh9 APmORNQZQH1gsBgsfLGMdXY4tCY/CITBgZjmetWkAQOJ98MvFniyGzZ9jCfv29rmJbym GeG4mcozIeZ6jOER3fKuVByTFou9p4ezpVgtmMEfyoB+nyhDGjXA6MifkrozZS75L1D5 vzBn/PAYaRr5383y41JHtPWyAYdDU+3I8ThF3FCuAqrene7IRm3XyJb6NWsEhvGhg426 jxLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1N+8R3B0R8scfqdo5hcRRcXMxMacRIFBiFcRxzf5Jso=; b=6U+/24hkVwRJqi9j1FyPEIHomX2Of6xpne0lW9Ix66rJBSVpEr1RVRcyA1bZ/Xhmcp VqDMfh/AjPDB80XNxWQ83ifLOgY1P+dtW+DJAtxGZsmdb6D9K0ZhSHuFTzKvgKJlCpec lsFOO8lUWgWeypkH79qXTYZZydwyeq468/UgChoVCrQXlQWg+Fvh0Por3STZmeSfGRIc mb4IzLS2SMFvfPvyJcg8EKS2HF8cPt1Kdr7wsv1pEKZ1fUcaPRSNZPZYEP7Z3EsLtXnF 6xk6T0y2bQqC38ciehhnSsq5ifvqZLa2JKQmtAaAJO4PzqDVlE5L+YTVAM7gdyVt3EIT XfsA== X-Gm-Message-State: ANoB5pmmwl5wVaYtJ8dlEbEQ6Ve4IxOGOhgYKOMMc4DoYhJYX2N1gh3z C1Lnj8BsPKnmpOcR3meiCrQ7Wg+1Yp0hHS2eNYStJw== X-Google-Smtp-Source: AA0mqf40PUHC3tnPKxX3qZepvqqyV1zyMITMf7TM43b3OWxLKuSILSQEYgxex+vVCemsw7KTP+Z5PZ6eKBQOI4Ffyfk= X-Received: by 2002:a05:6102:1c5:b0:3b0:4f9b:238d with SMTP id s5-20020a05610201c500b003b04f9b238dmr28776025vsq.59.1669946790569; Thu, 01 Dec 2022 18:06:30 -0800 (PST) MIME-Version: 1.0 References: <20221122203850.2765015-1-almasrymina@google.com> <874juonbmv.fsf@yhuang6-desk2.ccr.corp.intel.com> <87wn7dayfz.fsf@yhuang6-desk2.ccr.corp.intel.com> <87tu2e36nw.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: <87tu2e36nw.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Mina Almasry Date: Thu, 1 Dec 2022 18:06:18 -0800 Message-ID: Subject: Re: [RFC PATCH V1] mm: Disable demotion from proactive reclaim To: "Huang, Ying" Cc: Johannes Weiner , Yang Shi , Yosry Ahmed , Tim Chen , weixugc@google.com, shakeelb@google.com, gthelen@google.com, fvdl@google.com, Michal Hocko , Roman Gushchin , Muchun Song , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Queue-Id: 7CC8618000F X-Stat-Signature: mqsj19dry45jicys4fdm6k7c1qsduojz X-Rspamd-Server: rspam01 X-Spamd-Result: default: False [0.10 / 9.00]; SORBS_IRL_BL(3.00)[209.85.217.54:from]; BAYES_HAM(-3.00)[100.00%]; BAD_REP_POLICIES(0.10)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DMARC_POLICY_ALLOW(0.00)[google.com,reject]; RCPT_COUNT_TWELVE(0.00)[16]; DKIM_TRACE(0.00)[google.com:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; R_DKIM_ALLOW(0.00)[google.com:s=20210112]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[] X-HE-Tag: 1669946791-917282 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Dec 1, 2022 at 6:02 PM Huang, Ying wrote: > > Mina Almasry writes: > > > On Tue, Nov 29, 2022 at 7:56 PM Huang, Ying wrote: > >> > >> Johannes Weiner writes: > >> > >> > Hello Ying, > >> > > >> > On Thu, Nov 24, 2022 at 01:51:20PM +0800, Huang, Ying wrote: > >> >> Johannes Weiner writes: > >> >> > The fallback to reclaim actually strikes me as wrong. > >> >> > > >> >> > Think of reclaim as 'demoting' the pages to the storage tier. If we > >> >> > have a RAM -> CXL -> storage hierarchy, we should demote from RAM to > >> >> > CXL and from CXL to storage. If we reclaim a page from RAM, it means > >> >> > we 'demote' it directly from RAM to storage, bypassing potentially a > >> >> > huge amount of pages colder than it in CXL. That doesn't seem right. > >> >> > > >> >> > If demotion fails, IMO it shouldn't satisfy the reclaim request by > >> >> > breaking the layering. Rather it should deflect that pressure to the > >> >> > lower layers to make room. This makes sure we maintain an aging > >> >> > pipeline that honors the memory tier hierarchy. > >> >> > >> >> Yes. I think that we should avoid to fall back to reclaim as much as > >> >> possible too. Now, when we allocate memory for demotion > >> >> (alloc_demote_page()), __GFP_KSWAPD_RECLAIM is used. So, we will trigger > >> >> kswapd reclaim on lower tier node to free some memory to avoid fall back > >> >> to reclaim on current (higher tier) node. This may be not good enough, > >> >> for example, the following patch from Hasan may help via waking up > >> >> kswapd earlier. > >> >> > >> >> https://lore.kernel.org/linux-mm/b45b9bf7cd3e21bca61d82dcd1eb692cd32c122c.1637778851.git.hasanalmaruf@fb.com/ > >> >> > >> >> Do you know what is the next step plan for this patch? > >> >> > >> >> Should we do even more? > >> >> > >> >> From another point of view, I still think that we can use falling back > >> >> to reclaim as the last resort to avoid OOM in some special situations, > >> >> for example, most pages in the lowest tier node are mlock() or too hot > >> >> to be reclaimed. > >> > > >> > If they're hotter than reclaim candidates on the toptier, shouldn't > >> > they get promoted instead and make room that way? We may have to tweak > >> > the watermark logic a bit to facilitate that (allow promotions where > >> > regular allocations already fail?). But this sort of resorting would > >> > be preferable to age inversions. > >> > >> Now it's legal to enable demotion and disable promotion. Yes, this is > >> wrong configuration in general. But should we trigger OOM for these > >> users? > >> > >> And now promotion only works for default NUMA policy (and MPOL_BIND to > >> both promotion source and target nodes with MPOL_F_NUMA_BALANCING). If > >> we use some other NUMA policy, the pages cannot be promoted too. > >> > >> > The mlock scenario sounds possible. In that case, it wouldn't be an > >> > aging inversion, since there is nothing colder on the CXL node. > >> > > >> > Maybe a bypass check should explicitly consult the demotion target > >> > watermarks against its evictable pages (similar to the file_is_tiny > >> > check in prepare_scan_count)? > >> > >> Yes. This sounds doable. > >> > >> > Because in any other scenario, if there is a bug in the promo/demo > >> > coordination, I think we'd rather have the OOM than deal with age > >> > inversions causing intermittent performance issues that are incredibly > >> > hard to track down. > >> > >> Previously, I thought that people will always prefer performance > >> regression than OOM. Apparently, I am wrong. > >> > >> Anyway, I think that we need to reduce the possibility of OOM or falling > >> back to reclaim as much as possible firstly. Do you agree? > >> > > > > I've been discussing this with a few folks here. I think FWIW general > > feeling here is that demoting from top tier nodes is preferred, except > > in extreme circumstances we would indeed like to run with a > > performance issue rather than OOM a customer VM. I wonder if there is > > another way to debug mis-tiered pages rather than trigger an oom to > > debug. > > > > One thing I think/hope we can trivially agree on is that proactive > > reclaim/demotion is _not_ an extreme circumstance. I would like me or > > someone from the team to follow up with a patch that disables fallback > > to reclaim on proactive reclaim/demotion (sc->proactive). > > Yes. This makes sense to me. > Glad to hear it. Patch is already sent for review btw: https://lore.kernel.org/linux-mm/20221201233317.1394958-1-almasrymina@google.com/T/ > Best Regards, > Huang, Ying > > >> One possibility, can we fall back to reclaim only if the sc->priority is > >> small enough (even 0)? > >> > > > > This makes sense to me. > > > >> Best Regards, > >> Huang, Ying > >> >