From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43EEDC433ED for ; Fri, 30 Apr 2021 17:05:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C857F613EA for ; Fri, 30 Apr 2021 17:05:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C857F613EA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0258F6B006C; Fri, 30 Apr 2021 13:05:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F18B36B006E; Fri, 30 Apr 2021 13:05:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D926C6B0070; Fri, 30 Apr 2021 13:05:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0151.hostedemail.com [216.40.44.151]) by kanga.kvack.org (Postfix) with ESMTP id BCE056B006C for ; Fri, 30 Apr 2021 13:05:03 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6E4DC4DBF for ; Fri, 30 Apr 2021 17:05:03 +0000 (UTC) X-FDA: 78089658486.15.A8D1A49 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by imf06.hostedemail.com (Postfix) with ESMTP id 7F09CC0007F2 for ; Fri, 30 Apr 2021 17:05:05 +0000 (UTC) Received: by mail-wr1-f52.google.com with SMTP id h15so19119914wre.11 for ; Fri, 30 Apr 2021 10:05:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ef9fe3Au6iXOg8oECw1gAvJSq17/+FtuOxHdnbd2qQs=; b=cOM9YxXnZubuDxhIIhQclr+NWsfxbIv47SymfwxBPEOqNt9XlUcghOKU24KL1ydckx +STiiwZcwOExlo6OBZGoRcqvd34Q0mqMQVY3ioXlmC1LC3Ia65e5BccxlsBSvjlufLNK aDQmTS/AiPy0ihx5cHPRBESHL7AxHHs7e+LiPF57P1uZCnTC684leybf0VmVgjDtwngW s63XNL9xdaWeGl5/LKPWH6Y8vMlJiAn5/1j+vejyMwM6EjNktxViycRQE97XHwSi33jR Mj0+VNM47KdMw9iQDqfSaII216I24Ft+D2SIPJkw/Fs1BA+19FRhlYNx382HtGRQcSBz HTOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ef9fe3Au6iXOg8oECw1gAvJSq17/+FtuOxHdnbd2qQs=; b=S8ZxxgD0ZxBRjg43s1EikaiO9tj3J5L4wUkza2WIGA5Jmvalk9ssTZ6WKkWqnagkuy n2+igsAI3h0a6Tc53yN2wG5NKzhJ5Tx+93RKVpfiNZbEoWmaAcdJfXMwfrX8KPGXYTTJ D2cWqA+YRUIQ4nNi3ra+vB67FL0nnRXNrJe6JVG/flhahOG+1CWw2pZt6S9MqhxJQs94 nbLImM34SyyKz2omKHnRZBxJV4X0qCVTchu+kmXXChTsNOjwsiz2JdiaezIn/zrkrzaO Q/JiZIUXAP8uQBtl/FTsB1PlH8dMxLI5dfx0CHsRa4uhiKQPfLSAzubBREjFDtU0a/wy 64lg== X-Gm-Message-State: AOAM5330A0DFUlH4uv+3ATTeriesVVHwKOym7RpzeBbiRpsT7qwBrsnF ouW4Si5hIuU0m4aLGFXfRUGJ3SAG5Vwqr3s+CDGClQ== X-Google-Smtp-Source: ABdhPJzHYZmcpnQymDYuByc9ZujNal8ViIJt+704AIhqtiT8ouyvDBK9b7Mk4guwLGPFlDMUjbPrbfssdGFosS6oYWw= X-Received: by 2002:a05:6000:110d:: with SMTP id z13mr8394875wrw.92.1619802301471; Fri, 30 Apr 2021 10:05:01 -0700 (PDT) MIME-Version: 1.0 References: <20210416023536.168632-1-zhengjun.xing@linux.intel.com> <7b7a1c09-3d16-e199-15d2-ccea906d4a66@linux.intel.com> In-Reply-To: From: Yu Zhao Date: Fri, 30 Apr 2021 11:04:49 -0600 Message-ID: Subject: Re: [RFC] mm/vmscan.c: avoid possible long latency caused by too_many_isolated() To: Michal Hocko Cc: Xing Zhengjun , Andrew Morton , Linux-MM , linux-kernel , Huang Ying , Tim Chen , Shakeel Butt , wfg@mail.ustc.edu.cn, Rik van Riel , Andrea Arcangeli Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7F09CC0007F2 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=cOM9YxXn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of yuzhao@google.com designates 209.85.221.52 as permitted sender) smtp.mailfrom=yuzhao@google.com X-Stat-Signature: baohemb8wnsm9u67ecfnu9ots9w7ygtp Received-SPF: none (google.com>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=mail-wr1-f52.google.com; client-ip=209.85.221.52 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619802305-467968 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Apr 30, 2021 at 3:17 AM Michal Hocko wrote: > > On Fri 30-04-21 02:34:28, Yu Zhao wrote: > > On Thu, Apr 29, 2021 at 4:00 AM Michal Hocko wrote: > [...] > > > Still, I do not think that the above heuristic will work properly. > > > Different reclaimers have a different reclaim target (e.g. lower zones > > > and/or numa node mask) and strength (e.g. GFP_NOFS vs. GFP_KERNEL). A > > > simple count based throttling would be be prone to different sorts of > > > priority inversions. > > > > I see where your concern is coming from. Let's look at it from > > multiple angles, and hopefully this will clear things up. > > > > 1, looking into this approach: > > This approach limits the number of direct reclaimers without any bias. > > It doesn't favor or disfavor anybody. IOW, everyone has an equal > > chance to run, regardless of the reclaim parameters. So where does the > > inversion come from? > > Say you have a flood of GFP_KERNEL allocations contending with *MOVABLE > allocations. The former will not be able to reclaim for any non-kernel > zones. Similar effect would be contention of a heavy GFP_NOFS workload > condending with others but not being able to release filesystem > metadata. > > > 2, comparing it with the existing code: > > Both try to limit direct reclaims,: one by the number of isolated > > pages and the other by the number of concurrent direct reclaimers. > > Neither numbers are correlated with any parameters you mentioned above > > except the following: > > > > too_many_isolated() > > { > > ... > > /* > > * GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so they > > * won't get blocked by normal direct-reclaimers, forming a circular > > * deadlock. > > */ > > if ((sc->gfp_mask & (__GFP_IO | __GFP_FS)) == (__GFP_IO | __GFP_FS)) > > inactive >>= 3; > > ... > > } > > > > Let's at the commit that added the above: > > > > commit 3cf23841b4b7 ("mm/vmscan.c: avoid possible deadlock caused by > > too_many_isolated()"): > > Date: Tue Dec 18 14:23:31 2012 -0800 > > > > Neil found that if too_many_isolated() returns true while performing > > direct reclaim we can end up waiting for other threads to complete their > > direct reclaim. If those threads are allowed to enter the FS or IO to > > free memory, but this thread is not, then it is possible that those > > threads will be waiting on this thread and so we get a circular deadlock. > > > > some task enters direct reclaim with GFP_KERNEL > > => too_many_isolated() false > > => vmscan and run into dirty pages > > => pageout() > > => take some FS lock > > => fs/block code does GFP_NOIO allocation > > => enter direct reclaim again > > => too_many_isolated() true > > => waiting for others to progress, however the other > > tasks may be circular waiting for the FS lock.. > > > > Hmm, how could reclaim be recursive nowadays? > > I do not think it is. And I doubt it was back then and I also think the > above is not suggesting a recursion really. I tries to avoid a situation > when fs/block layer cannot make a fwd progress because it is being > blocked. > > > __alloc_pages_slowpath() > > { > > ... > > > > /* Avoid recursion of direct reclaim */ > > if (current->flags & PF_MEMALLOC) > > goto nopage; > > > > /* Try direct reclaim and then allocating */ > > page = __alloc_pages_direct_reclaim() > > ... > > } > > > > Let's assume it still could, do you remember the following commit? > > > > commit db73ee0d4637 ("mm, vmscan: do not loop on too_many_isolated for ever") > > Date: Wed Sep 6 16:21:11 2017 -0700 > > > > If too_many_isolated() does loop forever anymore, how could the above > > deadlock happen? IOW, why would we need the first commit nowadays? > > Whether the Neil's commit is still needed would require a deeper > analysis. Even back then we didn't perform pageout for fs dirty pages > from the direct reclaim IIRC. > > > If you don't remember the second commit, let me jog your memory: > > Yes i do remember that one and that was handling a dependency between > kswapd (which is allowed to perform pageout on diryt fs data) which > is blocked and it prevents direct reclaimers to make a fwd progress e.g. > by declaring OOM. This was mostly a band aid rather than a systematic > solution. And it clearly shows limits of the existing approach. Please > note that I am not trying to defend what we have now. I am just pointing > out that strict count based approach will hit other problems. > > > Author: Michal Hocko > > > > 3, thinking abstractly > > A problem hard to solve in one domain can become a walk in the park in > > another domain. This problem is a perfect example: it's different to > > solve based on the number of isolated pages; but it becomes a lot > > easier based on the number of direct reclaimers. > > This would be really true if all those reclaimers where equal in their > capabilities. But they are not due to reclaim constrains if nothing > else. Thanks for the clarification above. > IMHO the best way forward would be removing the throttling from the > reclaim path altogether. The reclaim should be only throttled by the > work it does. Both allocator and memcg charging path implement some sort > of retry logic and I believe this would be much better suited to > implement any backoff. I completely agree with this.